Form CMS-R-305 External Quality Review (EQR) Protocols

[Medicaid] External Quality Review (EQR) of Medicaid and Children’s Health Insurance Program (CHIP) Managed Care, EQR Protocols, and Supporting Regulations (CMS-R-305)

2023-eqr-protocols

(Private Sector) External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB: 0938-0786

Document [pdf]

Download: pdf | pdf

OMB Control No. 0938-0786
Expires: December 31, 2025
CMS External Quality Review (EQR) Protocols
February 2023
U.S. Department of Health and Human Services logo
Centers for Medicare & Medicaid Services (CMS) logo
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it
displays a valid OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required
to complete this information collection is estimated to average 7,688 hours per response for state governments and 185 hours per
response for the private sector, including the time to review instructions, search existing data resources, gather the data needed,
and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or
suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop
C4-26-05, Baltimore, Maryland 21244-1850.

This page left blank for double-sided copying.

CMS External Quality Review (EQR) Protocols Table
of Contents

Introduction ................................................................................................................................................... 1
Background ........................................................................................................................................... 1
Managed Care Quality Tools and the EQR Process ............................................................................ 3
Federal Financial Participation for EQR ................................................................................................ 7
Overview of the EQR Protocols ............................................................................................................ 8
Considerations Before Conducting EQR-Related Activities................................................................ 10
Nonduplication for Mandatory EQR-Related Activities ....................................................................... 11
Tips for Drafting Compliant EQR Technical Reports .......................................................................... 13
Tips for Drafting an Effective EQR Technical Report.......................................................................... 19
Tips for State Review of EQR Technical Reports ............................................................................... 20
Tips for Posting and Submitting EQR Technical Reports ................................................................... 20
Getting Started on the EQR Protocols ................................................................................................ 21
For Further Information ....................................................................................................................... 25
Protocol 1. Validation of Performance Improvement Projects .................................... 26
Background ......................................................................................................................................... 26
Getting Started on Protocol 1 .............................................................................................................. 27
Activity 1: Assess the PIP Methodology .............................................................................................. 28
Activity 2: Perform Overall Validation and Reporting of PIP Results .................................................. 38
Activity 3: Verify PIP Findings (Optional) ............................................................................................ 39
Worksheets for Protocol 1: PIP Validation Tools and Reporting Framework ..................................... 41

CONTENTS | iii

Protocol 2. Validation of Performance Measures ........................................................ 62
Background ......................................................................................................................................... 62
Getting Started on Protocol 2 .............................................................................................................. 63
Activity 1: Conduct Pre-Site Visit Activities ......................................................................................... 65
Activity 2: Conduct Site Visit Activities ................................................................................................ 74
Activity 3: Conduct Post-Site Visit Activities ........................................................................................ 80
Worksheets for Protocol 2: Performance Measure Validation Tools .................................................. 83
Protocol 3. Review of Compliance With Medicaid and CHIP Managed Care
Regulations ................................................................................................................. 125
Background ....................................................................................................................................... 125
Getting Started on Protocol 3 ............................................................................................................ 127
Activity 1: Establish Compliance Thresholds .................................................................................... 129
Activity 2: Perform the Preliminary Review (Pre-Site Visit) ............................................................... 130
Activity 3: Conduct MCP Site Visit .................................................................................................... 132
Activity 4: Compile and Analyze Findings (Post-Site Visit) ............................................................... 136
Activity 5: Report Results to the State............................................................................................... 138
Worksheets for Protocol 3: Compliance Review Tools ..................................................................... 140
Protocol 4. Validation of Network Adequacy ............................................................ 207
Background ....................................................................................................................................... 207
Getting Started on Protocol 4 ............................................................................................................ 210
Activity 1: Define the Scope of the Validation of Quantitative Network Adequacy Standards .......... 212
Activity 2: Identify Data Sources for Validation ................................................................................. 216
Activity 3: Review Information Systems Underlying Network Adequacy Monitoring......................... 217
Activity 4: Validate Network Adequacy Assessment Data, Methods, and Results ........................... 219
Activity 5: Communicate Preliminary Findings to Each MCP............................................................ 222
Activity 6: Submit Findings to State ................................................................................................... 222
Worksheets For Protocol 4: Network Adequacy Validation Tools..................................................... 224

iv | CONTENTS

Protocol 5. Validation of Encounter Data Reported by the Medicaid and CHIP
Managed Care Plan ..................................................................................................... 239
Background ....................................................................................................................................... 239
Getting Started on Protocol 5 ............................................................................................................ 240
Activity 1: Review State Requirements ............................................................................................. 242
Activity 2: Review the MCP’s Capability............................................................................................ 243
Activity 3: Analyze Electronic Encounter Data .................................................................................. 246
Activity 4: Review Medical Records .................................................................................................. 252
Worksheets for Protocol 5: Encounter Data Tables .......................................................................... 255
Protocol 6. Administration or Validation of Quality of Care Surveys ........................ 267
Background ....................................................................................................................................... 267
Getting Started on Protocol 6 ............................................................................................................ 268
Section I. Administering a Survey ............................................................................................................. 271
Activity I.1: Identify the Survey Purpose, Objectives, and Audience ................................................ 271
Activity I.2: Develop a Work Plan ...................................................................................................... 272
Activity I.3: Select the Survey Instrument.......................................................................................... 272
Activity I.4: Develop the Sampling Plan ............................................................................................ 276
Activity I.5: Develop a Strategy to Maximize Response.................................................................... 278
Activity I.6: Develop a Quality Assurance Plan ................................................................................. 281
Activity I.7: Implement the Survey According to the Work Plan ........................................................ 281
Activity I.8: Prepare and Analyze Survey Data and Present Results in a Final Report .................... 282
Section II. Validating a Survey .................................................................................................................. 285
Activity II.1: Review the Survey Purpose, Objectives, and Audience ............................................... 285
Activity II.2: Review the Work Plan ................................................................................................... 286
Activity II.3: Review the Reliability and Validity of the Survey Instrument ........................................ 286
Activity II.4: Review the Sampling Plan ............................................................................................. 287
Activity II.5: Review the Adequacy of the Response Rate ................................................................ 288
Activity II.6: Review the Quality Assurance Plan .............................................................................. 288
Activity II.7: Review the Survey Implementation ............................................................................... 289
CONTENTS | v

Activity II.8: Review the Survey Data Analysis and Final Report ...................................................... 289
Technical Appendix for Protocol 6: Understanding Potential Sources of Survey Error .................... 290
Worksheets for Protocol 6: Survey Administration and Validation Tools .......................................... 293
Protocol 7. Calculation of Additional Performance Measures ................................... 302
Background ....................................................................................................................................... 302
Getting Started on Protocol 7 ............................................................................................................ 303
Activity 1: Prepare for Measurement ................................................................................................. 304
Activity 2: Calculate Measures .......................................................................................................... 306
Activity 3: Report Results .................................................................................................................. 310
Worksheets for Protocol 7: Performance Measure Calculation Tools .............................................. 312
Protocol 8. Implementation of Additional Performance Improvement Projects ....... 319
Background ....................................................................................................................................... 319
Getting Started on Protocol 8 ............................................................................................................ 320
Activity 1: Select the PIP Topic ......................................................................................................... 322
Activity 2: Define the PIP Aim Statement .......................................................................................... 323
Activity 3: Identify the PIP Population ............................................................................................... 323
Activity 4: Use Sound Sampling Methods ......................................................................................... 324
Activity 5: Select the PIP Variables ................................................................................................... 324
Activity 6: Collect Valid and Reliable Data ........................................................................................ 325
Activity 7: Analyze Data and Interpret Results .................................................................................. 325
Activity 8: Review Improvement Strategies ....................................................................................... 326
Activity 9: Assess Whether Significant and Sustained Improvement Occurred ................................ 326
Protocol 9. Conducting Focus Studies of Health Care Quality .................................. 328
Background ....................................................................................................................................... 328
Getting Started on Protocol 9 ............................................................................................................ 329
Activity 1: Select the Study Topic(s).................................................................................................. 330
Activity 2: Define the Study Question(s)............................................................................................ 331
Activity 3: Select the Study Variable(s) ............................................................................................. 331

vi | CONTENTS

Activity 4: Develop a Plan to Study the Population ........................................................................... 334
Activity 5: Collect Data ...................................................................................................................... 335
Activity 6: Analyze and Interpret Study Results ................................................................................ 336
Activity 7: Report Results to the State............................................................................................... 337
Protocol 10. Assist With Quality Rating of Medicaid and CHIP Managed Care
Organizations, Prepaid Inpatient Health Plans, and Prepaid Ambulatory Health Plans
.................................................................................................................................... 338
Reserved. .......................................................................................................................................... 338
Appendix A. Information Systems Capabilities Assessment ......................................... 1
Background ........................................................................................................................................ A.1
Getting Started on the ISCA ............................................................................................................... A.4
Activity 1: MCP Completes the ISCA Tool ......................................................................................... A.5
Activity 2: Perform Preliminary ISCA Review ..................................................................................... A.5
Activity 3: Conduct MCP Onsite or Virtual Site Visit .......................................................................... A.6
Activity 4: Compile and Analyze ISCA Findings ................................................................................. A.6
Activity 5: Draft ISCA Summary for EQR Technical Report ............................................................... A.7
Worksheets for Appendix A ................................................................................................................ A.9
Appendix B. Sampling Approaches for EQR Data Collection Activities ........................ 1
Background ........................................................................................................................................ B.1
Types of Sampling Approaches ......................................................................................................... B.1
Calculating Minimum Sample Sizes for EQR Data Collection Activities ............................................ B.3
Documenting Sampling Methods for EQR Data Collection Activities ................................................ B.5
Appendix C. Acronyms Used In the Protocols ............................................................... 1
Appendix D. External Quality Review Glossary of Terms .............................................. 1

CONTENTS | vii

Worksheets
Worksheet 1.1. Review the Selected PIP Topic ......................................................................................... 43
Worksheet 1.2. Review the PIP Aim Statement.......................................................................................... 45
Worksheet 1.3. Review the Identified PIP Population ................................................................................ 46
Worksheet 1.4. Review the Sampling Method ............................................................................................ 47
Worksheet 1.5. Review the Selected PIP Variables and Performance Measures ..................................... 48
Worksheet 1.6. Review the Data Collection Procedures ............................................................................ 51
Worksheet 1.7. Review Data Analysis and Interpretation of PIP Results ................................................... 54
Worksheet 1.8. Assess the Improvement Strategies .................................................................................. 55
Worksheet 1.9. Assess the Likelihood that Significant and Sustained Improvement Occurred ................. 57
Worksheet 1.10. Perform Overall Validation of PIP Results ....................................................................... 58
Worksheet 1.11. Framework for Summarizing Information about Performance Improvement
Projects (PIPs) ............................................................................................................................... 59
Worksheet 2.1. List of Performance Measures to be Validated.................................................................. 85
Worksheet 2.2. Performance Measure Validation Template ...................................................................... 86
Example of Worksheet 2.2. Performance Measure Validation Template ................................................... 90
Worksheet 2.3. Medical Record Review Validation Template .................................................................... 95
Worksheet 2.4. Potential Documents and Processes for Review............................................................... 99
Worksheet 2.5. Interview Guide for MCP Data Integration and Control Personnel .................................. 104
Worksheet 2.6. Data Integration and Control Findings Tool ..................................................................... 106
Worksheet 2.7. Data and Processes Used to Produce Performance Measures: Documentation
Review Checklist .......................................................................................................................... 108
Worksheet 2.8. Data and Processes Used to Produce Performance Measures: Findings ...................... 110
Worksheet 2.9. Policies, Procedures, and Data Used to Produce Performance Measures: Review
Checklist ....................................................................................................................................... 112
Worksheet 2.10. Measure Validation Findings ......................................................................................... 114
Worksheet 2.11. Interview Guide for Assessing Processes and Procedures Used to Produce
Numerators and Denominators .................................................................................................... 117
Worksheet 2.12. Policies, Procedures, and Data Used to Implement Sampling: Review Checklist ........ 118
Worksheet 2.13. Sampling Validation Findings ........................................................................................ 119
Worksheet 2.14. Framework for Summarizing Information About Performance Measures ..................... 121
viii | CONTENTS

Worksheet 3.1. Compliance Review ......................................................................................................... 141
Worksheet 3.2. Compliance Definitions .................................................................................................... 162
Worksheet 3.3. Sample Site Visit Agenda ................................................................................................ 164
Worksheet 3.4. Compliance Interview Questions ..................................................................................... 165
Worksheet 4.1. State Network Adequacy Standards to be Validated ...................................................... 225
Worksheet 4.2. Network Adequacy Indicators to be Validated ................................................................. 226
Worksheet 4.3. Data Sources for Network Adequacy Validation.............................................................. 227
Worksheet 4.4. Network Adequacy Data Concerns Identified in Review of ISCA.................................... 229
Worksheet 4.5. Assessment of Network Adequacy Data Sources not Reviewed in the ISCA................. 230
Worksheet 4.6. Assessment of MCP Network Adequacy Data, Methods, and Results ........................... 231
Worksheet 4.7. Summary of Network Adequacy Validation Findings ...................................................... 236
Worksheet 4.8. Recommendations to Improve MCP Assessment of Network Adequacy ....................... 238
Worksheet 5.1. Specification of Acceptable Error Rates and Identified Areas of Concern ...................... 256
Worksheet 5.2. Data Element Validity Requirements ............................................................................... 257
Worksheet 5.3. Evaluation of Submitted Fields ........................................................................................ 258
Worksheet 5.4. Benchmark Utilization Rates............................................................................................ 259
Worksheet 5.5. Medical Record Review for Encounter Data Validation................................................... 261
Worksheet 5.6. Medical Record Review Results Summary Sheet ........................................................... 263
Worksheet 5.7. Suggested Format for Reporting Encounter Data Validation Information in the
EQR Technical Report ................................................................................................................. 264
Worksheet 6.1. Survey Purpose, Objectives, and Audience .................................................................... 294
Worksheet 6.2. Work Plan ........................................................................................................................ 295
Worksheet 6.3. Survey Instrument ............................................................................................................ 296
Worksheet 6.4. Sampling Plan .................................................................................................................. 297
Worksheet 6.5. Strategy to Maximize Response ...................................................................................... 298
Worksheet 6.6. Quality Assurance Plan ................................................................................................... 299
Worksheet 6.7. Survey Implementation According to the Work Plan ....................................................... 300
Worksheet 6.8. Survey Data Analysis and Final Report ........................................................................... 301
Worksheet 7.1. List of Performance Measures to be Calculated.............................................................. 313
Worksheet 7.2. Companion Performance Measurement Tool.................................................................. 314

CONTENTS | ix

Worksheet 7.3. Data Element Master Checklist ....................................................................................... 316
Worksheet 7.4. Data Availability and Data Quality ................................................................................... 317
Worksheet 7.5. Illustrative File Format for Transmission of Claims Data ................................................. 318
Worksheet A.1. Information System Capabilities Assessment (ISCA) Tool .............................................. A.9
Worksheet A.2. Information System Review Worksheet & Interview Guide............................................ A.36

x | CONTENTS

Introduction
Background
Together, Medicaid and the Children’s Health Insurance Program (CHIP) cover more than 83
million people, 1 representing about 1 in 4 people in the United States, and covering 42 percent of
all births. 2 Nationally, 69 percent of adults and children enrolled in Medicaid and CHIP obtain
their care through managed care plans (MCPs), although the rate of managed care enrollment in
states using a managed care delivery system varies widely. 3 The federal requirements related to
Medicaid managed care quality, including the external quality review (EQR) process, were
established in statute at section 1932(c) of the Social Security Act (the Act) and are set forth in
42 CFR 438(c). The same statutory federal requirements were made applicable to CHIP managed
care quality through section 2103(f)(3) of the Act and are set forth in 42 CFR 457.1240 and
1250. Box 1 defines key terms related to the EQR process.
Box 1. Key Definitions Related to the External Quality Review Process
• Managed care plan (MCP). Encompasses managed care organizations (MCOs), prepaid inpatient health plans
(PIHPs), prepaid ambulatory health plans (PAHPs), and primary care case management (PCCM) entities described
in 42 CFR 438.310(c)(2).
• External quality review (EQR). EQR is the analysis and evaluation of aggregated information on quality,
timeliness, and access to the health services that an MCP or its contractors furnish to Medicaid beneficiaries [see
42 CFR 438.320]. EQR can only be conducted by a qualified EQRO.
• External quality review organization (EQRO). An EQRO is an organization that meets the competence and
independence requirements set forth in 42 CFR 438.354, and performs EQR, EQR-related activities, or both.
• EQR-related activities. The activities addressed in these protocols. EQR-related activities produce the data used
by an EQRO to complete the annual EQR. EQR-related activities may be conducted by the state, its agent that is
not an MCP, or an EQRO [see 42 CFR 438.358]..

The Centers for Medicare & Medicaid Services (CMS) published the Medicaid and CHIP
managed care final rule in May 2016, which aligns key rules with those of other health insurance
coverage programs, modernizes how states purchase managed care for beneficiaries, and
strengthens the consumer experience and key consumer protections. 4 The rule also updated and
expanded EQR in the following ways:

Estimates are for July 2021. July 2021 Medicaid and CHIP Enrollment Data Highlights. Available at
https://www.medicaid.gov/medicaid/program-information/medicaid-and-chip-enrollment-data/report-highlights/index.html.

Data on births covered by Medicaid and CHIP in 2019 is available at https://www.cdc.gov/nchs/data/nvsr/nvsr70/nvsr70-02508.pdf.

Data on the percentage of Medicaid beneficiaries in comprehensive managed care, by state, is available at
https://www.medicaid.gov/medicaid/quality-of-care/downloads/beneficiary-profile-2021.pdf.

More information about the Medicaid and CHIP managed care final rule is available at
https://www.medicaid.gov/medicaid/managed-care/guidance/final-rule/index.html and the Federal Register at
https://www.gpo.gov/fdsys/pkg/FR-2016-05-06/pdf/2016-09581.pdf.

INTRODUCTION | 1

•

Clarified that the Children’s Health Insurance Program Reauthorization Act of 2009
(CHIPRA) applied EQR (including EQR-related activities) to both separate CHIP MCPs and
Medicaid Expansion CHIP MCPs. A state that uses MCPs to provide CHIP benefits must
develop and implement a managed care quality strategy and must require CHIP MCPs to
operate quality assessment and performance improvement (QAPI) programs

•

Applied EQR to a broader range of Medicaid MCPs, that is, beyond managed care
organizations (MCOs) and prepaid inpatient health plans (PIHPs) to also include prepaid
ambulatory health plans (PAHPs) and primary care case management (PCCM) entities whose
contracts with the state provide for shared savings, incentive payments or other financial
reward for the PCCM entity for improved quality outcomes 5

•

Added two EQR-related activities: (1) a mandatory EQR-related activity, validation of
network adequacy (effective no later than one year from the issuance of the associated EQR
protocol) and (2) an optional EQR-related activity, assistance with the quality rating of
MCOs, PIHPs, and PAHPs required under a Medicaid and CHIP quality rating system
(effective no earlier than the issuance of the associated EQR protocol) 6

In November 2020, CMS released revisions to the 2016 final rule. The 2020 rule updated the
following EQR provisions:
•

Required states to identify MCOs exempt from EQR annually on its website in the same
location where EQR technical reports are posted. State must include the name(s) of the
MCO(s) exempt from EQR, including the beginning date of the current exemption period, or
that no MCOs are exempt from EQR.

•

Clarified standards for the EQR compliance review activity in 42 CFR 438.358(b)(1)(iii) by
referencing the full set of subpart B, C, and D standards that comprise the compliance
review. The rule made further modifications to the standards subject to EQR in each of these
subparts, and CMS encourages states and EQROs to familiarize themselves with these
changes.

•

Inserted several technical revisions to CHIP regulations that cross-reference Medicaid EQR
standards to align CHIP and Medicaid EQR standards, and those quality standards relevant to
EQR reporting requirements.

The timeline in Figure 1 chronicles the evolution of the scope of EQR in Medicaid and CHIP.
These updated protocols reflect changes included in the 2020 rule.

For the purposes of the EQR protocols, all references to PCCM entities should be assumed to refer to the applicable subset of
PCCM entities described at 42 CFR 438.310(c)(2), and 457.1240(f).

Page 27499 of the 2016 final rule includes effective dates for the additional EQR-related activities: “States must begin
conducting the EQR-related activity described in 438.358(b)(1)(iv) (relating to the mandatory-EQR-related activity of the
validation of network adequacy) no later than one year from the issuance of the associated EQR-protocol. States may begin
conducting the EQR-related activity described in 438.358(c)(6) (relating to the optional EQR-related activity of plan rating) no
later than the issuance of the associated EQR protocol.”

2 | INTRODUCTION

Figure 1. Evolution of EQR in Medicaid and CHIP

a Balanced

Budget Act of 1997 amending section 1932(c)(1)(B) of the Social Security Act.
1139A(c)(2) of the Social Security Act, as amended by section 401(a) of CHIPRA, requires the U.S.
Department of Health & Human Services (HHS) Secretary to summarize State-specific information on the quality of
health care furnished to children under titles XIX (Medicaid) and XXI (CHIP). Section 1139A(c)(1)(B) of the Act
specifically requests information gathered from the external quality reviews of managed care organizations (MCOs)
and benchmark plans.
b Section

Managed Care Quality Tools and the EQR Process
EQR is one part of a multipronged approach to Medicaid and CHIP managed care quality. These
managed care quality tools are interrelated, with each informing and reinforcing the others
(Figure 2). For example, state quality strategies articulate managed care priorities, including
goals and objectives for quality improvement. QAPI programs should reflect the priorities
articulated in the quality strategy and include specific measures and targets from the quality
strategy, with performance improvement projects (PIPs) aimed at driving improvement on the
measures. The performance measures and PIPs are then validated during the annual EQR, with
results included in the EQR technical report. The EQR technical report also includes
recommendations from the EQRO on how states can target quality strategy goals and objectives
to support improvements in quality of care, as well as reviews the standards in 42 CFR 438.56,
438.110, 438.114, 42 CFR subpart D, and 42 CFR 438.330. Additionally, in the 2022 revision of
the EQR protocols, Validation of Network Adequacy is being added as an essential oversight
component for managed care plans. By thinking holistically about these managed care quality
INTRODUCTION | 3

components (see Figure 2 for the interrelationships among QS, QAPI, and EQR), states can
maximize the impact of their managed care quality initiatives.
Figure 2. Relationship between State Medicaid and CHIP Managed Care Quality Initiatives

Figure 3 outlines the EQR process. States using a managed care delivery system for all or some
of their Medicaid and/or CHIP beneficiaries are required to contract with a qualified independent
EQRO to conduct an annual EQR to assess and monitor the quality of care provided to Medicaid
and CHIP beneficiaries enrolled in MCPs and to identify opportunities for quality improvement. 7
To simplify the narrative of these protocols, the term “EQRO” is used to refer to the entity which
conducts the EQR-related activities that generate the information for the annual EQR. An EQRO
is the only entity which may conduct the annual EQR, that is, the analysis and evaluation of
information generated by the EQR-related activities (or via nonduplication, if applicable)
regarding the quality, timeliness, and access to the health care services that an MCP, or its
contractors, furnish to beneficiaries. States with both Medicaid and CHIP managed care
programs may elect to contract with a single EQRO to conduct EQR of both Medicaid and CHIP
or may contract with different EQROs for EQR of Medicaid and CHIP. Many states choose to
utilize the same EQRO for EQR of both Medicaid and CHIP.

CHIP regulations at 42 CFR 457.1250 cross-reference to the Medicaid managed care EQR requirements at 42 CFR 438.356.

4 | INTRODUCTION

The end product of the EQR is an annual EQR technical report, which summarizes findings on
access and quality of care, and must be drafted by said EQRO. 8
Figure 3. The EQR Process

The EQR process also includes a series of mandatory and optional EQR-related activities
designed to provide a sound understanding of the strengths and weaknesses of Medicaid and
CHIP MCP performance related to quality, timeliness, and access to care (see Box 2). The EQRrelated activities are intended to (1) improve states’ ability to oversee and manage the MCPs they
contract with for services, and (2) help MCPs improve their performance with respect to quality,
timeliness, and access to care. Effective implementation of the EQR-related activities will
facilitate state efforts to purchase high-value care (rather than volume) and to achieve higher
performing health care delivery systems for their Medicaid and CHIP beneficiaries. States have
flexibility regarding who will conduct the EQR-related activities; they may be conducted by the
state, its agent that is not a managed care plan, or an EQRO. If the state elects to contract with an
EQRO to conduct the EQR-related activities, this can be the same EQRO that conducts the EQR
for the state, or one or more additional EQROs. 9

CHIP regulations at 42 CFR 457.1250 cross-reference to Medicaid managed care EQR requirements at 42 CFR 438.364.

States may choose to contract with different entities, including more than one EQRO, for different EQR-related activities. For
example, the state might validate performance improvement projects (see Protocol 1) itself, contract with EQRO A for the
validation of performance measures (see Protocol 2), and contract with EQRO B for the compliance review (see Protocol 3).
Said state could then contract with EQRO A, B, or a third EQRO (C) to conduct the EQR and produce the EQR technical
report. For information on state contracting options for EQR, see 42 CFR 438.356 (as cross referenced at 457.1250 for CHIP.

INTRODUCTION | 5

Box 2. Mandatory and Optional EQR-Related Activities
Mandatory EQR-Related Activities
• Validate performance improvement projects (PIPs)
• Validate performance measures
• Review compliance with Medicaid and CHIP managed care regulations
• Validate network adequacy (Reserved)
Optional EQR-Related Activities
• Validate encounter data reported by MCPs
• Administer or validate quality of care surveys
• Calculate additional performance measures
• Conduct additional PIPs
• Conduct focus studies of health care quality
• Assist with quality rating of MCPs (Reserved)

Medicaid and CHIP MCOs, PIHPs, and PAHPs are subject to all four mandatory EQR-related
activities; 10 PCCM entities are subject to two of the mandatory EQR-related activities (a
compliance review and validation of performance measures). 11 See Table 1 for additional
information about the application of EQR-related activities to MCPs.
Table 1. Application of Mandatory and Optional EQR-related Activities by MCP Type
MCP Type
EQR-Related Activity

MCO

PIHP

PAHP

PCCM Entity

Validation of Performance Improvement Required
Projects

Required

State Discretion

Validation of Performance Measures

Required

Review of Compliance with Medicaid
Managed Care Regulations

Required

Validation of Network Adequacya

Required

N/A

Validation of Encounter Data Reported
by the MCP

State Discretion

Administration or Validation of Quality
of Care Surveys

State Discretion

The network adequacy validation requirement will go into effect one year after the release of the network adequacy validation
protocol. Until then, MCOs, PIHPs, and PAHPs will be subject to three mandatory EQR-related activities: Protocol 1
(Validation of PIPs), Protocol 2 (Validation of Performance Measures), and Protocol 3 (Review of Compliance with Medicaid
Managed Care Regulations) (Figure 4).

While regulations do not require PCCM entities to conduct PIPs as a part of their QAPI programs, states may choose to
require their PCCM entities to do so. States that require PCCM entities to conduct PIPs should consider validating those PIPs.

6 | INTRODUCTION

MCP Type
EQR-Related Activity

MCO

PIHP

PAHP

PCCM Entity

Calculation of Additional Performance
Measures

State Discretion

Implementation of Additional
Performance Improvement Projects

State Discretion

Conducting Focus Studies of Health
Care Quality

State Discretion

Assist with Quality Rating of Medicaid
and CHIP MCOs, PIHPs, and PAHPsa

State Discretion

N/A

a States

may not claim for these EQR-related activities until the EQR protocol is issued.

Federal Financial Participation for EQR
For Medicaid programs, EQR (including the production of the EQR technical report) and EQRrelated activities performed on MCOs, as well as the production of the EQR technical report are
eligible for Federal financial participation (FFP) at a 75 percent match rate (1) when conducted
by a qualified EQRO, (2) when the EQR-related activities are completed using methodologies
consistent with the protocols contained within this document, and (3) when the state receives
approval of its EQRO contract from CMS. 12, 13 The EQRO’s analysis is eligible for the 75
percent match rate when the information from a Medicare or private accreditation review of an
MCO is used for the mandatory EQR-related activities. However, the accreditation activities that
produce the information cannot receive the match.
Medicaid programs are eligible for the 50 percent match rate if an agency other than a qualified
EQRO conducted the EQR-related activities. 14 EQR (including the production of the EQR
technical report) and EQR-related activities conducted on PIHPs, PAHPs and PCCM entities are
eligible for the 50 percent match rate. 15 For CHIP, EQR and EQR-related activities are subject to
the 10 percent administrative cap as required by section 2105(c)(2)(A) of the Act; a state is

See 42 CFR 433.15 and 438.370(a) and the July 10, 2016 CMCS Informational Bulletin (CIB), Federal Financial Participation
for Managed Care External Quality Review, available at https://www.medicaid.gov/federal-policyguidance/downloads/cib061016.pdf.

If the state or the state’s agent that is not an MCP conducts the EQR-related activity on an MCO, it would be eligible for the
50 percent match rate. See 42 CFR 438.370(a)–(b). When information from a Medicare or private accreditation review of an
MCO is used to support one or more mandatory EQR-related activities in place of a Medicaid review, the EQRO’s analysis of
the MCO data as part of the EQR is eligible for FFP at the 75 percent rate. The accreditation activities that produce the
information are not eligible for the FFP.

See 42 CFR 433.15 and 438.370(b).

See 42 CFR 438.370(b). Note that this is a change from the previous regulation, under which the enhanced match was
available for EQR of PIHPs to the same extent as MCOs. For further explanation of the change, see discussion in the Medicaid
and CHIP Managed Care 2016 final rule at 81 CFR 27498, 27715-27716.

INTRODUCTION | 7

eligible to receive the state’s enhanced CHIP FFP match rate for these activities, regardless of
which entity completes the activity.

Overview of the EQR Protocols
CMS is required to develop EQR protocols to guide and support the annual EQR process. 16 The
first set of protocols was issued in 2003, updated in 2012 (recall Figure 1), and updated again in
2019 to incorporate regulatory changes contained in the May 2016 Medicaid and CHIP managed
care final rule. The 2019 revised protocols were also designed to improve the user experience
navigating through the components, provide new tools to drive improvement using current
industry methodologies (such as rapid cycle evaluation approaches), and offer practical tips and
best practices for reporting.
CMS is required to review the protocols and make necessary revisions every three years. In
2022, CMS issued the third revision to the EQR Protocols to incorporate regulatory changes
contained in the 2020 final rule, clarify federal requirements for the EQR process to promote
compliance, respond to state and EQRO feedback about the protocols, and include the network
adequacy validation protocol.
Figure 4 (next page) identifies the EQR protocols linked to each of the mandatory and optional
EQR-related activities, as well as the source of the regulations that guide the protocols. In
addition, an Information Systems Capabilities Assessment (ISCA) is a mandatory component of
the EQR as part of Protocols 1, 2, 3, and 4, as well as Protocols 5 and 7 (if applicable). Box 3
shows the general content of the EQR protocols.
Box 3. Content of the EQR Protocols
• Purpose of the EQR-related activity
• How to conduct each activity within the protocol, including:
○ Data sources and data collection activities to promote data accuracy, validity, and reliability
○ Proposed method(s) for analyzing and interpreting the data
○ Instructions, guidelines, worksheets, and/or tools that may be used in implementing the protocol

See section 1932(c)(2)(A)(iii) of the Social Security Act and 42 CFR 438.352.

8 | INTRODUCTION

Figure 4. Overview of the EQR Protocols

Notes:

CHIP regulations at 42 CFR 457.1250(a) cross-reference to all the Medicaid managed care EQR
requirements at 42 CFR 438.320.

It should be noted that several protocols are organized around site visits by the EQRO. In
recognition of changing practices related to the COVID-19 pandemic, site visits may be
conducted onsite or virtually to obtain the information specified in the protocols.
INTRODUCTION | 9

States must ensure that the privacy of patient information is protected in a manner consistent
with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) 17 throughout all
EQR-related activities and the EQR technical report process. Specifically, the 2016 final rule
introduced requirements that EQR technical reports do not disclose a patient’s identify or any
Protected Health Information (PHI). 18 Consistent with that obligation, states should ensure that
their MCPs comply with HIPAA and all other federal and state laws concerning confidentiality
and disclosure. The EQRO should ensure that its EQR-related data collection and reporting
activities meet these requirements.
The next section of this introductory chapter discusses practical considerations for states before
beginning the EQR-related activities. The following section provides tips to guide the drafting of
effective EQR technical reports that document performance in regard to quality, timeliness, and
access to care; identify areas for improvement; and recommend interventions to improve the
process and outcomes of care. Links to the protocols and appendices are contained at the end of
this chapter. The four appendices are: Information Systems Capabilities Assessment (Appendix
A), Sampling Approaches for EQR Data Collection Activities (Appendix B), Acronyms Used in
the Protocols (Appendix C), and External Quality Review Glossary of Terms (Appendix D).

Considerations Before Conducting EQR-Related Activities
Preparing to conduct EQR-related activities involves several steps (see Box 4). EQR-related
activities may be performed by the state, an agent of the state that is not an MCP, or by an
EQRO. 19 These protocols are applicable to EQR-related activities conducted by any of these
entities. While most states hire an EQRO to conduct the EQR-related activities, states may elect
to conduct the EQR-related activities themselves or to contract with an organization that is not an
EQRO or an agent that is not an MCP to perform these activities. Regardless of which entity
performs EQR-related activities, the data from all activities must be independently reviewed and
evaluated by the EQRO as part of the annual EQR and EQR technical report.

See 42 CFR 431 Subpart F and 457.1110.

See 42 CFR 438.364(d).

See 42 CFR 438.358(a).

10 | INTRODUCTION

Box 4. Steps to Prepare for EQR-Related Activities
1 Select an entity to conduct the EQR-related activity(s)
○ Ensure that staff conducting EQR-related activities have the training and experience needed for the particular
activity(s) they will be conducting
2 Provide clear, written understanding of the parameters of the review
○ List of MCPs for review
○ Select optional EQR-related activities (if applicable) in addition to the applicable mandatory EQR-related
activities
○ Designate a timeframe for review
3 Review all applicable federal regulations, state regulations or standards, and MCP state contracts
4 Confirm approach with entity and all EQR participants
○ Each organization's responsibilities in collecting, reporting, and/or analyzing data
○ Which regulations, contracts, and/or initiatives should be evaluated
○ Which reviews will occur and tools used
○ A timeline identifying the start and completion of each protocol

Nonduplication for Mandatory EQR-Related Activities
Nonduplication is intended to reduce administrative burden on MCPs and states while still
ensuring relevant information is available to EQROs for the annual EQR. The expansion of
nonduplication to three of the mandatory EQR-related activities (Protocols 1–3, Validation of
Performance Improvement Projects, Validation of Performance Measures, and Review of
Compliance with Medicaid Managed Care Regulations) 20 for all Medicaid managed care MCOs,
PIHPs, and PAHPs—not just those serving only dually eligible beneficiaries—provides
additional flexibility to states to reduce administrative burden. Nonduplication is an option for a
state only when the Medicare or accreditation review standards are comparable to the EQR
protocols (not vice versa). If a state elects to use nonduplication, it must document in its
managed care quality strategy and its annual EQR technical report the EQR-related activities for
which it utilizes nonduplication along with the state’s rationale for its determination that the
Medicare or private accreditation review standards are comparable to those in these protocols. 21
The federal requirements related to nonduplication of mandatory activities are described in 42
CFR 438.360. Like Medicaid, CHIP MCPs may submit information from a private accreditation
review; however, with regard to CHIP, information documenting compliance with Medicare
Advantage standards is not applicable as described in 42 CFR 457.1250(a).
Nonduplication allows a state to use information from a Medicare or private accreditation review
of an MCP in place of generating that information through one or more of three mandatory EQR-

Nonduplication is not an option for the fourth mandatory EQR-related activity of network adequacy validation (42 CFR
438.358(b)(1)(iv)).

See 42 CFR 438.360(c) and 438.340(b)(10).

INTRODUCTION | 11

related activities (Protocols 1–3, Validation of Performance Improvement Projects, Validation of
Performance Measures, and Review of Compliance with Medicaid Managed Care
Regulations). 22 To do so, the following conditions must be met:
•

The MCP is in compliance with the applicable Medicare Advantage or private accreditation
standards 23

•

The Medicare or private accreditation review standards are comparable to those established
through the EQR protocols for the three mandatory EQR-related activities

•

The MCP provides the state with all applicable reports, findings, and other results of the
Medicare or private accreditation review applicable to the specified EQR-related activities

The state is responsible for providing the EQRO with all information from the Medicare or
private accreditation review which is being used for nonduplication. The EQRO then assesses the
completeness of information from the accreditation review to determine the extent of
nonduplication, including confirming the comparable information fully meets the requirements
for completing the analysis and developing EQR findings and recommendations. If a state
chooses nonduplication, it must ensure the completion of any EQR-related activities (or
components of those activities) that are not addressed by the information from the Medicare or
private accreditation review. For example, if an accreditation review did not validate long term
services or supports (LTSS) or other non-Healthcare Effectiveness Data and Information Set
(HEDIS®) measures required by the state as a part of an MCP’s QAPI program, that validation
activity would need to be completed for those measures.
It is important to note that even when information from a Medicare or private accreditation
review does not completely meet the requirements of an activity, that information can still be
used toward meeting the nonduplication requirements. For example, nonduplication might be
able to satisfy a subset of the regulatory requirements that are subjects of the compliance review.
In this example, the EQRO could use information from the nonduplication source for that subset
of requirements, and then the EQR-related activity would only need to be conducted on the
remaining requirements to fully assess compliance. Similarly, if a state requires its MCPs to
include 10 measures in QAPI and 5 are validated as a part of an accreditation review, only the
other 5 would need to be validated through the EQR-related activity. Validation information on
all 10 measures would then be provided to the EQRO for the EQR.

Prior to issuance of the Medicaid and CHIP 2016 final rule, such information could only be used to provide information which
would otherwise be gathered from performing the mandatory EQR-related compliance review.

See 42 CFR 422 subpart D.

12 | INTRODUCTION

Box 5. What is the Difference Between Nonduplication and Exemption?
Nonduplication is a way to provide information for the annual EQR without conducting part of, or all of, one or more
EQR-related activities by using information yielded by a comparable review process. Under nonduplication, an MCO,
PIHP, or PAHP is still subject to EQR and will be included in the annual EQR technical report. Nonduplication may be
used at the state’s discretion and consistent with documentation in the state’s managed care quality strategy.
Exemption is an option which allows a state to exempt an MCO (but not a PIHP or PAHP) from the annual EQR
process under certain circumstances. If a state exempts an MCO from EQR, the MCO will not be included in the
annual EQR technical report. Exemption may be used at the state’s discretion when the following three conditions are
met:
• The MCO has both a current Medicare Advantage contract and a current Medicaid contract;
• The two contracts cover all or part of the same geographic area in the state; and
• The Medicaid contract has been in effect for at least two consecutive years before the exemption date, and during
those same two years, the MCO has been subject to EQR and met quality, timeliness, and access to health care
services standards for Medicaid beneficiaries
If a state wants to exempt an MCO from EQR, it must obtain either of the following:
• For MCOs reviewed by Medicare, the state must obtain annually the most recent Medicare review findings from the
MCO, including all data, correspondence, information, and findings relevant to the MCO’s compliance with
Medicare standards for (1) access, quality assessment and performance improvement, health services, or
delegation of these activities, (2) all measures of the MCO’s performance, and (3) results and findings of all
performance improvement projects for Medicare enrollees
• For MCOs reviewed by a private, national accrediting organization that CMS approves and recognizes for Medicare
Advantage Organization deeming, the state must require that the MCO provide a copy of all findings from its most
recent accreditation review if that review was used to meet certain requirements for Medicare external review, or to
determine compliance with Medicare requirements. At a minimum, findings must include accreditation review
results of evaluation of compliance with individual accreditation standards, any deficiencies, corrective action plans,
and summaries of unmet accreditation requirements
Each year, the state must identify MCOs exempt from EQR on its website in the same location where EQR technical
reports are posted. The state must include the name(s) of the exempt MCO(s), including the beginning date of the
current exemption period, or that no MCOs are exempt from EQR.
Complete requirements for exemption of MCOs are available at 42 CFR 438.362.

When information from a Medicare or private accreditation review of an MCP is used to support
one or more mandatory EQR-related activities, the EQRO’s analysis of the data is eligible for
FFP. The accreditation activities that produce the information are not eligible for the FFP. Note
that use of discretion of the state, not its MCPs. See Box 5 for more information on the
difference between nonduplication and exemption.

Tips for Drafting Compliant EQR Technical Reports
A qualified EQRO 24 is the only entity that may conduct the annual EQR, that is, the analysis and
evaluation of information generated by the EQR-related activities (or via nonduplication, if

See 42 CFR 438.354 for information about the competence and independence requirements for an EQRO.

INTRODUCTION | 13

applicable) regarding the quality, timeliness, and access to the health care services that an MCP,
or its contractors, furnish to beneficiaries. The end product of the EQR is an EQR technical
report, which must be drafted by said EQRO for the state. 25 This section provides guidance on
report content and structure.

Guidance on Report Content
EQROs should produce reports that meet all federal requirements. To promote compliance with
federal requirements, this section summarizes the requirements and includes considerations for
drafting EQR technical reports. Table 2 provides an overview of the required elements in EQR
technical reports and Table 3 identifies requirements for the PIP validation (Protocol 1),
performance measure validation (Protocol 2), and review of compliance activities (Protocol 3).

CHIP regulations at 457.1250 cross reference to 42 CFR 438.364.

14 | INTRODUCTION

Table 2. Required Elements in EQR Technical Reports and Tips for Drafting EQR Technical Reports
Regulatory Reference

Requirement

Tips for Drafting EQR Technical Reports

42 CFR 438.364(a)

All eligible Medicaid and CHIP plans are included in the report.

Identify the MCPs subject to EQR by plan name, MCP type,
managed care authority, and population(s) served in an introduction,
executive summary, or appendix.

42 CFR 438.364(a)(1)

The technical report must summarize findings on quality, access,

Include a description for all three activities noted under the regulation
(1) how data were aggregated, (2) how they were was analyzed, and
(3) how conclusions were drawn about the MCP’s ability to furnish
services. These findings should reflect a comparison to the domains
of quality, timeliness, and access to healthcare services..

42 CFR 438.364(a)(3)

The technical report must include an assessment of the
strengths and weaknesses of each MCO, PIHP, PAHP and
PCCM entity with respect to (a) quality, (b) timeliness, and (c)
access to the health care services furnished by MCO's, PIHP's,
PAHP's, or PCCM entity.

Include a chart outlining each MCP’s strengths and weaknesses for
each EQR activity and designate a quality, timeliness, and access
domain

The technical report must include recommendations for
improving the quality of health care services furnished by each
MCO, PIHP, PAHP, or PCCM entity.

Include recommendations for each MCP. Recommendations should
share the EQRO’s understanding of why the weakness exists and
suggest steps for how the MCP—potentially in concert with the
state—can best address the issue. If the cause for the weakness is
unclear or unknown, the EQRO should suggest how the MCP and/or
state can identify the cause.

42 CFR 438.364(a)(4)

and timeliness of care for each MCO, PIHP, PAHP, and PCCM entity
that provides benefits to Medicaid and CHIP enrollees.

Highlight substantive findings concerning the extent to which each
MCP is furnishing high quality, timely, and appropriate access to
health care services. Findings should focus on the specific strengths
and weaknesses the EQRO identified, rather than on numerical
ratings or validation scores obtained under the EQRO’s review
methodology.

When determining recommendations, EQROs should consider
whether the suggested actions are within the authority of the MCP
(or state).

INTRODUCTION | 15

Regulatory Reference

Requirement

Tips for Drafting EQR Technical Reports

42 CFR 438.364(a)(4)

The technical report must include recommendations for how the
state can target goals and objectives in the quality strategy,
under §438.340, to better support improvement in the quality,
timeliness, and access to health care services furnished to
Medicaid or CHIP beneficiaries.

Consider connecting EQR findings to the quality strategy goals and
objectives, particularly in sections of the report that assess the
state’s overall performance of the quality, timeliness, and access to
health care services; when discussing strengths and weaknesses of
a MCP or activity; or when discussing the basis of performance
measures or PIPs. Note when goals in the quality strategy are
considered in EQR activities and which goals they are. Describe the
relationship between goals in the state’s quality strategy and the four
mandatory EQR activities.

42 CFR 438.364(a)(5)

The technical report must include methodologically appropriate, Aggregate findings across MCPs for each EQR activity, and show
comparative information about all MCOs, PIHPs, PAHPs, and comparisons.
PCCM entities.
Provide context for the individual MCP to make it easier for
stakeholders to understand the results of the review and more readily
determine whether issues are localized or systemic.

42 CFR 438.364(a)(6)

The technical report must include an assessment of the degree
to which each MCO, PIHP, PAHP, or PCCM entity has
addressed effectively the recommendations for quality
improvement made by the EQRO during the previous year’s
EQR.

State the prior year finding and describe the assessment of each
MCP’s approach to addressing the recommendation or findings
issued by the state or EQRO in the previous year’s EQR technical
report. This is not a restatement of a response or rebuttal to the
recommendation by the MCP or state.
Document assessments with the same specificity used when
reporting on initial findings.

42 CFR 438.364(d)

The information included in the technical report must not
Ensure the technical report is consistent with the Health Insurance
disclose the identity or other protected health information of any Portability and Accountability Act of 1996 (HIPAA) (42 C.F.R. §431
patient.
Subpart F and § 457.1110).
Ensure that MCPs comply with HIPAA and all other federal and state
laws concerning confidentiality and disclosure.
Ensure that EQR-related data collection and reporting activities are
consistent with HIPAA requirements.

16 | INTRODUCTION

Table 3. Requirements for the PIP Validation, Performance Measure Validation, and Review of Compliance Activities
and Considerations for Drafting EQR Technical Reports
Regulatory Reference
42 CFR 438.364(a)(2)(iiv)

Requirement
The technical report must include the following for each of the
mandatory activities
•

Objectives

•

Technical methods of data collection and analysis

•

Description of data obtained including validated
performance measurement data for each PIP

•

Conclusions drawn from the data.

Considerations for Drafting EQR Technical Reports
Objectives: Provide the state or EQRO’s objective for conducting
the mandatory activity itself, including the general approach or
methods of validation used by the EQRO. The state may also include
the objective or aim statement for each PIP to satisfy this criteria for
the PIP validation activity.
Technical methods of data collection and analysis: Provide a
description of how data was obtained by the EQRO to conduct the
validation activity. If a collection tool is used, providing an example
of the format of the tool, or questions asked, in an appendix is a best
practice. Further, describe how data is analyzed to connect the data
requested to the analytical methods that eventually support the
conclusions drawn with those data and analyses.
Description of data obtained: Based upon the collection efforts
above, describe the types of data obtained – information system
extracts, documents, answers to questions in data collection tools,
and others – to explain the nature of the data collected and analyzed.
Note: This requirement does not apply to the compliance review
activity (Protocol 3).
Conclusions drawn from the data: Having employed the process
of data collection and validation using the types and nature of the
data received, provide conclusions relevant to the mandatory activity.

438.358(b)(1)(i)

The technical report must include information on the validation
of PIPs that were underway during the preceding 12 months.

Provide a validation of all PIPs underway during the 12 month period
preceding the EQR review, regardless of the phase of the PIP’s
implementation. States often link the time-frame under review to a
corresponding measurement or performance period such as state or
federal fiscal year, or calendar year.

42 CFR 438.330(d)

The technical report must include a description of PIP
interventions associated with each state-required PIP topic for
the current EQR review cycle.

For states with many MCPs and PIPs, provide an appendix or link to
each plan-level report, an appendix in an aggregate report, or a
separate PIP-report that compiles the PIPs applicable to all or a
group of plans. Present this information in a cohesive way that allows
for brevity in the sections that describe data analysis and
conclusions.

INTRODUCTION | 17

Regulatory Reference

Requirement

Considerations for Drafting EQR Technical Reports
Note that a table listing all PIP interventions will not alone be
considered as methodologically appropriate comparative information,
as the table simply organizes information, but does not compare or
draw conclusions from the information presented.

42 CFR 438.358(b)(1)(ii)

42 CFR 438.358(b)(1)(iii)

The technical report must include information on the validation
of each MCO’s, PIHP’s, PAHP’s, or PCCM entity’s performance
measures for each MCO, PIHP, PAHP, and PCCM entity
performance measure calculated by the state during the
preceding 12 months.
The technical report must include information on a review,
conducted within the previous three-year period, to determine
each MCO's, PIHP's, PAHP's or PCCM’s compliance with the
standards set forth in Subpart D and the QAPI requirements
described in 42 CFR 438.330.

States often link the time-frame under review to a corresponding
measurement or performance period such as state or federal fiscal
year, or calendar year.

For each of the 10 Subpart D standards and individual QAPI
standard, ensure that the method of compliance review clearly links
the EQRO’s activities to the standard under review. Further, ensure
that a clear compliance determination is made and recorded for each
standard for each plan. A best practice is to list a compliance score
The technical report must provide MCP results for the following of a numerical or semi-quantitative nature.
11 Subpart D and QAPI standards:
EQROs that assess domains, standards, and requirements that do
not neatly overlap with the regulatory standards should provide a
42 CFR 438.206, 457.1230(a), Availability of services
clear crosswalk of their activities to the standards under review. As a
42 CFR 438.207, 457.1230(b), Assurances of adequate
best practice, the technical report may include a table outlining the
capacity and services
timeline for reviewing all standards for MCPs across the three-year
42 CFR 438.208, 457.1230(c) Coordination and continuity of
review period.
care
42 CFR 438.210, 457.1230(d), Coverage and authorization of
services
42 CFR 438.214, 457.1233(a), Provider selection
42 CFR 438.224, 457.1230(c), Confidentiality
42 CFR 438.228, 457.1260, Grievance and appeals system
42 CFR 230, 457.1233(b), Subcontractual relationships and
delegation
42 CFR 438.236, 457.1233(c), Practice guidelines
42 CFR 438.242, 457.1233(d), Health information system
42 CFR 438.330, 457.1240(b), QAPI

18 | INTRODUCTION

Provide a validation of all performance measures in use during the
12 month period preceding the EQR review, regardless of the phase
of the performance measure’s implementation.

Guidance on Report Structure
To be of greatest use to states and other stakeholders, EQROs should draft reports that are
actionable, clear, and concise; that highlight substantive findings; and that contain actionable
recommendations. CMS understands that states vary in the number of Medicaid and CHIP
programs and MCPs. The EQRO should prepare an aggregate report that summarizes results
across all MCPs and provides state-level recommendations for performance improvement. The
EQR technical reports must also meet the plan-level reporting requirements for items such as
identification of strengths and weaknesses, assessment of MCP actions to address previous
year’s recommendations, performance measures, and others. The EQRO can address these planlevel reporting requirements via tables or appendices to the aggregate report or prepare separate
aggregate reports by type of MCP if appropriate. For example, the EQRO may develop one
aggregate report on all of a state’s medical MCOs and a separate aggregate report on all of the
state’s behavioral health PIHPs. See Box 6 for guidance on using aggregate and plan-level
reports.
Box 6. Using Aggregate and Plan-level Reports
States with large and varied managed care programs may group MCPs by type, geography, or populations served
and provide both plan-level and aggregate reports. Plan-level reports may be effective for communicating planspecific information while aggregate reports can communicate required comparative information and analyses.
Creating multiple reports may help CMS and the public find and review information quickly, but states and EQROs
should describe the structure of reports in a statewide executive summary or in the introduction to the aggregate
report.

Tips for Drafting an Effective EQR Technical Report
In addition to focusing on compliance with federal requirements, EQROs should consider these
tips for drafting a report that is actionable, clear, and concise.
•

Aim for clarity and concise presentation. While every EQR review necessarily gathers and
processes a substantial amount of material, non-essential narrative makes it difficult for
readers to identify the most relevant information

Because not all readers have deep experience in the areas covered by EQR, avoid
technical language and jargon when possible

To maximize interpretability of results, provide context for all statistics included in the
report

•

Include a clickable or hyperlinked table of contents. For easy navigation throughout the
report(s), include a clickable or hyperlinked table of contents

•

Produce a searchable PDF. To enable stakeholders to review topics of interest and facilitate
use of the reports for topic-specific analyses, produce a searchable PDF
INTRODUCTION | 19

•

Use the names of MCPs when referring to plan performance. Findings and comparisons
should refer to MCPs by name in order to facilitate transparency and stakeholder
understanding of specific plan performance

•

Consider displaying previous recommendations, plan responses and actions, and new
recommendations in one chart. This enables a comprehensive view of the history of each
MCP’s performance based on the EQR process

The comparative information should include tables presenting, for all plans,
performance measure scores, and PIP ratings and scores

Charts can be used to display compliance and non-compliance with each of the reviewed
state and federal standards

Tips for State Review of EQR Technical Reports
The state is ultimately responsible for the submission of a complete report per 42 CFR 438.364.
Upon receipt of EQR technical reports from the EQRO, the state should review the report for
completeness and adherence to these protocols. For example, the state should confirm that the
technical report includes all of the required elements set forth in 42 CFR 438 Subpart E and
includes a review of all standards and regulations in Subpart E of Part 438 and other regulations
incorporated by reference.
The EQR technical report must be independently developed and produced by the EQRO. The
state may not substantively revise the EQR technical report submitted by the EQRO without
evidence that errors or key omissions occurred.

Tips for Posting and Submitting EQR Technical Reports
As required under 42 CFR 438.10(c)(3), the state must post its finalized annual technical
report(s) on its website by April 30th of each year. CMS additionally requests states submit their
EQR technical report(s) to [email protected] by April 30th of each year.
Box 7 provides reminders to states before posting and submitting their EQR technical reports.
Box 7. Reminders to States
• For the purposes of the EQR protocols, we refer to EQROs as the entity conducting the EQR-related activities.
While most states hire an EQRO to conduct the EQR-related activities, the state may elect to conduct the EQRrelated activities themselves or to contract with an EQRO or an agent that is not an MCP to perform these activities
(or a combination of these approaches). An EQRO is the only entity that may conduct the annual EQR. The end
product of the EQR is an EQR technical report, which must be drafted by said EQRO.
• An accrediting body may not serve as an EQRO for a health plan it accredited within the previous 3 years.
• States cannot substantively revise the EQR technical report without evidence that errors occurred or that key
information was omitted.

20 | INTRODUCTION

Getting Started on the EQR Protocols
So far, this chapter has provided background on the mandatory and optional EQR-related
activities and the associated protocols, discussed considerations before conducting EQR-related
activities, provided an overview of nonduplication, and shared tips on drafting the EQR technical
report. Now it’s time to review the protocols for each of the activities and begin planning the
approach to conducting the EQR. Use the “Go Now!” buttons to navigate to the individual
protocols and the appendices.
The protocols in this document are designed to support the completion of the EQR-related
activities, which in turn help the state meet the requirement to conduct an EQR of its MCPs and
help contracted EQROs meet the requirements of producing an annual EQR technical report. If a
state prefers to use methods consistent with but not identical to these protocols to conduct EQRrelated activities, the state is encouraged to discuss the alternative methods with CMS before
implementation to assure the methods meet regulatory standards.
If you have any questions related to the EQR protocols or alternative methods, please contact
CMS via the TA mailbox, [email protected].

INTRODUCTION | 21

Mandatory EQR-Related Activities
Protocol 1 – Validation of Performance Improvement Projects

Page 26

MCOs, PIHPs, and PAHPs are required to implement performance improvement projects (PIPs)
that focus on both clinical and non-clinical aspects of care. Protocol 1 specifies procedures for
EQROs to use in assessing the validity and reliability of a PIP (42 CFR 438.358(b)(i)).
Protocol 2 – Validation of Performance Measures

Page 62

MCPs must report standard performance measures as specified by the state. The state must
provide to the EQRO and the MCP the performance measures to be calculated, the specifications
for the measures, and the state reporting requirements. Protocol 2 tells the EQRO how to:
•

Evaluate the accuracy of the Medicaid and CHIP MCP reported performance measures based
on the measure specifications and state reporting requirements

•

Evaluate if the MCP followed the rules outlined by the state agency for calculating the
measures (42 CFR 438.358(b)(ii))

This protocol also applies when a state requires its MCPs to submit data to the state so that the
state can calculate the standard performance measures
Protocol 3 – Review of Compliance with Medicaid and CHIP
Managed Care Regulations

Page 125

The EQR is required to include a compliance review of each MCP once in a 3-year period.
Protocol 3 specifies procedures to determine the extent to which MCPs comply with standards
set forth at 42 CFR 438.358(b)(iii), state standards, and MCP contract requirements.
Note that states may meet the 3-year requirement in different ways: for example, some review all
MCPs at the same time once every 3 years; others conduct a complete compliance review on a
subset of plans each year on a 3-year cycle. While a full compliance review is required for each
MCP once every 3 years, the state must address any EQR findings in the next reporting year.
Protocol 4 – Validation of Network Adequacy

Page 210

States must ensure that Medicaid and CHIP MCPs maintain provider networks that are sufficient
to provide timely and accessible care to Medicaid and CHIP beneficiaries across the continuum
of services. As set forth in 42 CFR 438.68, states are required to set quantitative network
adequacy standards for MCPs that account for regional factors and the needs of the state’s
Medicaid and CHIP populations.
22 | INTRODUCTION

The purpose of this protocol is to guide the EQRO in conducting the validation of network
adequacy during the preceding 12 months to comply with requirements set forth in 42 CFR
438.68 and, if the state enrolls American Indians and Alaska Natives in the MCO, PIHP, or
PAHP, 42 CFR 438.14(b)(1). This includes validating data to determine whether the network
standards, as defined by the state, were met.

Optional EQR-Related Activities
Protocol 5 – Validation of Encounter Data Reported by the MCP

Page 242

The states use managed care encounter data, which are data about a distinct service provided to
an enrollee, to better understand the health services delivered by the MCP, assess and review
quality, monitor program integrity, and determine capitation payment rates. Protocol 5 specifies
procedures for assessing the completeness and accuracy of encounter data submitted by MCPs to
the state. It also assists in the improvement of processes associated with the collection and
submission of encounter data from MCPs to the state.
Protocol 6 – Administration or Validation of Quality of Care Surveys Page 271

Surveys are a common method of measuring health care quality, especially consumer experience
with care. Protocol 6 specifies procedures for conducting various types of surveys and validating
those surveys.
Protocol 7 – Calculation of Additional Performance Measures

Page 306

The state uses performance measures to monitor the performance of MCPs over time, to
understand the MCPs’ impact on the Medicaid population, to compare the performance of
different MCPs, and to inform the selection and evaluation of quality improvement activities.
Protocol 7 specifies procedures for calculating MCP performance measures in accordance with
the state specifications. It also supplies information to the state on the extent to which the MCP’s
information system provides accurate and complete information necessary for the calculation of
performance measures.
Protocol 8 – Implementation of Additional Performance
Improvement Projects

Page 323

The state may conduct—or request an EQRO conduct—a PIP in addition to those MCOs, PIHPs,
and PAHPs are required to conduct as a part of their QAPI programs. Protocol 8 specifies
procedures for implementing additional PIPs.

INTRODUCTION | 23

Protocol 9 – Conducting Focus Studies of Health Care Quality

Page 332

The state may choose to conduct a study on a particular aspect of clinical and/or non-clinical
services provided by its MCPs. Protocol 9 specifies procedures to plan and carry out a focus
study.
Protocol 10 – Assist with the Quality Rating of Medicaid and CHIP MCOs, PIHPs,
and PAHPs

Page 342

[Reserved]

Appendices
The EQR protocol package includes four appendices to supplement information contained in the
protocols. Use the “Go Now!” buttons to navigate to the appendices.
Appendix A. Information Systems Capabilities Assessment

Page A.1

Protocols 1, 2, 3, 4, 5, and 7 require each state to assess their MCPs’ information system (IS)
capabilities. The regulations at 42 CFR 438.242 and 457.1233(d) also require the state to ensure
that each MCP maintains a health information system that collects, analyzes, integrates, and
reports data for areas including, but not limited to, utilization, grievances and appeals, and
disenrollments for reasons other than the loss of Medicaid eligibility. Portions of the Information
Systems Capabilities Assessment (ISCA) are voluntary; however, there are components that
relate directly to the mandatory EQR-related activity protocols. It defines the recommended
capabilities of an MCP’s information system to meet the above noted regulatory requirements, as
well as how to assess the strength of the MCP’s information system capabilities. It includes an
overview of the processes for collecting, processing, and reporting data, and guidance for:
•

Completing the ISCA assessment (by MCPs)

•

Reviewing ISCA and accompanying documents

•

Interviewing MCP staff

•

Analyzing ISCA findings

Appendix B. Sampling Approaches for EQR Data Collection
Activities

Page B.1

This appendix provides an overview of sampling approaches that can be used in Protocols 1, 2,
5, 6, 7, 8, and 9.

24 | INTRODUCTION

Appendix C. Acronyms Used in the Protocols

Page C.1

This appendix defines acronyms used in the protocols.
Appendix D. External Quality Review Glossary of Terms

Page D.1

This appendix defines terms used in the Protocols.

For Further Information
Technical assistance resources related to EQR, including the EQR protocols, are available on
Medicaid.gov at https://www.medicaid.gov/medicaid/quality-of-care/medicaid-managedcare/external-quality-review/index.html.
Please submit any questions or requests for technical assistance related to EQR to
[email protected].

INTRODUCTION | 25

Protocol 1. Validation of Performance
Improvement Projects
A Mandatory EQR-Related Activity

ACTIVITY 2: PERFORM OVERALL VALIDATION AND REPORTING OF
PIP RESULTS
ACTIVITY 3: VERIFY PIP FINDINGS (OPTIONAL)

Background
States must require their Medicaid and CHIP managed care plans (MCPs) to
conduct performance improvement projects (PIPs) that focus on both clinical
and nonclinical areas each year as a part of the plan’s quality assessment and
performance improvement (QAPI) program, per 42 CFR 438.330 and
457.1240(b). 26 See Box 1.1 for the definition of a PIP.
Box 1.1. What is a PIP?
A PIP is a project conducted by the MCP that is designed to achieve significant
improvement, sustained over time, in health outcomes and enrollee experience. A PIP may
be designed to change behavior at a member, provider, and/or MCP/system level. The topic
should target improvement in relevant areas of clinical and non-clinical services.

This external quality review (EQR)-related activity validates the PIPs that the
MCP was required to conduct as part of its QAPI program. The external quality
review organization (EQRO) reviews the PIP design and implementation using
documents provided by the MCP, which may be supplemented with interviews
of MCP staff. The EQRO then reports to the state on its findings from reviewing
and validating the PIP(s) in the EQR technical report. As noted in the
Introduction, states have the option to use information from a Medicare or

PROTOCOL ONE

ACTIVITY 1: ASSESS THE PIP METHODOLOGY

At a minimum, a single PIP that focuses on both clinical and non-clinical aspects of care may satisfy this
requirement. Otherwise, a state must require at least two PIPs, one clinical and one non-clinical.

private accreditation review of an MCP to provide information for the annual EQR instead of
conducting this mandatory EQR-related activity. 27, 28
A related protocol, Protocol 8. Implementation of Additional Performance Improvement Projects,
specifies procedures for implementing additional PIPs in accordance with state specifications.

Getting Started on Protocol 1
To complete this protocol, the EQRO undertakes two required activities and one optional activity
for validating the PIPs for each MCP (Figure 1.1).
Figure 1.1. Protocol 1 Activities

If the state elects to use nonduplication for this mandatory EQR-related activity (42 CFR 438.360, Nonduplication of mandatory
activities with Medicare or accreditation review), then the state must ensure that all information from the Medicare or private
accreditation review is provided to the EQRO for analysis and inclusion in the annual EQR technical report. (See 42 CFR
438.360(a)(1)–(3) for additional details regarding the circumstance under which nonduplication is an option). Use of nonduplication
must be identified in the state’s quality strategy (see 42 CFR 438.360(c) and 438.340(b)(10)). CHIP regulations at 457.1250 crossreferences to 42 CFR 438.360, but does not allow for the use of Medicare review activities for the purposes of nonduplication.

A state may not utilize nonduplication if Medicare has accepted an only attestation of a plan’s quality incentive program (QIP). In
the context of this EQR-related activity, the QIP would have to undergo validation as part of a Medicare review in order for
nonduplication to be an option. See 42 CFR 438.360(a)(2).

PROTOCOL ONE | 27

Two supplemental resources are available to help EQROs validate PIPs, including:
•

Worksheets for Protocol 1. PIP Validation Tools and Reporting Framework, a set of
worksheets that can be used to guide and record answers for the validation of PIPs and
reporting of summary PIP information, based on activities 1 through 3 and associated steps in
this protocol

•

Appendix B. Sampling Approaches for EQR Data Collection Activities, which provides an
overview of sampling methods that could be used in this protocol

The remainder of this protocol outlines the steps associated with Activities 1 through 3.

Activity 1: Assess the PIP Methodology
The EQRO should complete the nine steps in Activity 1, listed below, and answer the questions
posed in each step.

Step 1: Review the Selected PIP Topic
PIP topics should target improvement in relevant areas of
clinical and non-clinical services. In this step, the EQRO
determines the appropriateness of the PIP topic(s), including
how the PIP topic was selected and input from enrollees or
providers. CMS suggests that PIP topics align with CMSidentified priorities 29 and consider the aims of the National
Quality Strategy: 30
•

Better care for patients and families

•

Improved health for communities and populations

•

Affordable health care

WORKSHEET 1.1
Resource for Activity 1, Step 1
Worksheet 1.1. Review the Selected PIP
Topic
• Provides a template for assessing the
appropriateness of the PIP topic,
including how the PIP topic was
selected, the consideration of the
CMS Child and Adult Core Set
measures, and input from enrollees
or providers

More information about CMS priorities and initiatives is available at https://www.cms.gov/Medicare/Quality-InitiativesPatient-Assessment-Instruments/Value-Based-Programs/CMS-Quality-Strategy

More information about the National Quality Strategy is available at
https://www.ahrq.gov/workingforquality/about/index.html.

28 | PROTOCOL ONE

In addition, the state should review its performance on the CMS Child and Adult Core Set
measures 31 to identify opportunities to improve performance through a managed care PIP.

Step 2: Review the PIP Aim Statement
In this step, the EQRO assesses the appropriateness
and adequacy of the aim statement. The PIP aim
statement identifies the focus of the PIP and
establishes the framework for data collection and
analysis. The PIP aim statement should define the
improvement strategy, population, and time period. It
should be clear, concise, measurable, and answerable.
Box 1.2 identifies considerations for developing a
PIP aim statement and Table 1.1 provides a critique
of illustrative PIP aim statements.

WORKSHEET 1.2
Resource for Activity 1, Step 2
Worksheet 1.2. Review the PIP Aim
Statement
• Provides a template for reviewing the
PIP Aim Statement

Box 1.2. Considerations for Developing a PIP Aim Statement that is Clear, Concise, Measurable,
and Answerable
A PIP aim statement is clear, concise, measurable, and answerable if the statement specifies measurable variables
and analytics for a defined improvement strategy, population, and time period. Potential sources of information to help
form the PIP aim statement include:
• State data relevant to the topic being studied
• MCP data relevant to the topic being studied
• CMS Child and Adult Core Set measures
• Enrollee focus groups or surveys
• Relevant clinical literature on recommended care and external benchmarks

Table 1.1. Critique of Example PIP Aim Statements
Example PIP Aim Statements
Poor PIP
Aim
Statement

Does the MCP adequately address psychological
problems in patients recovering from myocardial
infarction?

Critique
• The PIP intervention is not specified
• It is unclear how impact will be measured
• The population and time period are not
clearly defined

More information about the Child and Adult Core Sets available at https://www.medicaid.gov/medicaid/quality-ofcare/performance-measurement/adult-and-child-health-care-quality-measures/index.html.

PROTOCOL ONE | 29

Example PIP Aim Statements
Good PIP
Aim
Statement

Critique

Will the use of cognitive behavioral therapy in patients • Specifies the PIP intervention (cognitive
with depression and obesity improve depressive
behavioral therapy)
symptoms over a six-month period during 2022?
• Defines the population (patients with
depression and obesity) and the time period
(six-month period during 2022)
• Specifies the measurable impact (improve
depressive symptoms)

Step 3. Review the Identified PIP Population
In this step, the EQRO assesses whether the MCP clearly
WORKSHEET 1.3
identified the population for the PIP in relation to the PIP
aim statement (such as age, length of enrollment,
Resource for Activity 1, Step 3
diagnoses, procedures, and other characteristics).
Worksheet 1.3. Review the Identified
Depending on the nature of the PIP aim statement, PIP
PIP Population
population, and available data, the PIP may include the
• Provides a template for assessing
entire population or a sample of the population. PIPs that
whether the PIP population was
rely on existing administrative data, such as claims and
appropriately identified
encounter data, registry data, or vital records, are typically
based on the universe of the PIP population. PIPs that rely on either medical record review or
the hybrid method (which uses a combination of administrative data and medical record review)
typically include a representative sample of the identified population. If a sample was used for
the PIP, go to Step 4. If the entire population was studied, skip Step 4 and go to Step 5. If
HEDIS® measures and sampling methodology are used, go to Step 5.

Step 4: Review the Sampling Method
In this step, the EQRO assesses the appropriateness of the
PIP’s sampling methods. Appropriate sampling methods
are necessary to ensure that the collection of information
produces valid and reliable results. Please refer to
Appendix B, Sampling Approaches for EQR Data
Collection Activities, for an overview of sampling
methodologies applicable to PIPs. When HEDIS®
measures are used and sampling is required (for example,
for measures calculated using the hybrid method),
HEDIS® sampling methodology should be used.

30 | PROTOCOL ONE

WORKSHEET 1.4
Resource for Activity 1, Step 4
Worksheet 1.4. Review the Sampling
Method
• Provides a template for reviewing the
suitability of the sampling method
based on the PIP aim statement and
population

Step 5: Review the Selected PIP Variables and Performance Measures
In this step, the EQRO assesses the variables selected for a
WORKSHEET 1.5
PIP (see Box 1.3). Variables in PIPs can take a variety of
forms as long as the selected variables identify the MCP’s
Resource for Activity 1, Step 5
performance on the PIP questions objectively and reliably
Worksheet 1.5. Review the Selected PIP
and use clearly defined indicators of performance. The PIP
Variables and Performance Measures
should include the number and types of variables that are
• Provides a template for assessing the
adequate to answer the PIP question and for which
appropriateness of selected PIP
appropriate and reliable data are available to measure
variables and performance measures
performance and track improvement over time. Variables
used in PIPs may be continuous, categorical, or discrete (Table 1.2), and may use a variety of
measurement scales to assess performance (Table 1.3).
Box 1.3. Considerations for Selecting Variables for a PIP
• A variable is a measurable characteristic, quality, trait, or attribute of a particular individual, object, or situation
being studied (see Table 1.2 for types of variables).
• When choosing variables, consider different types of variables and choose the variables and measurement scales
that are best suited to the available data, resources, and PIP aim statement (see Table 1.3 for types of measure
scales).
• CMS encourages MCPs to choose variables that reflect health outcomes or that can be linked to health outcomes
and that can be examined on at least a semi-annual basis.

Table 1.2. Types of Variables for PIPs
Variable Type
Continuous

Definition
Have a range of numerical values
Note: Data collected for a continuous variable can be
recoded as a discrete variable (e.g., an enrollee’s
blood pressure is above or below a specified level)

Examples
• Age, blood pressure, temperature,
height/weight, body mass index,
birthweight

Categorical

Have a range of non-ordered, qualitative values (or
categories)

• An enrollee survey question that asks
enrollees to identify the most important
among a list of incentives offered to
improve well-care visit rates

Discrete

Have a limited number of possible categories

• An enrollee has/has not received a flu
shot in the past 12 months

Note: binary variables have two categories

PROTOCOL ONE | 31

Table 1.3. Types of Measurement Scales for PIP Variables
Measurement
Scales

Definition

Example

Interval

The distances between numbers denote significant
and interpretable differences (e.g., dollars, degrees,
inches, pounds) and the differences are interpretable
as higher or lower.

• The interval between an annual income
of $40,000 and $30,000 = $10,000

Ordinal

Can be treated as quantitative in some
circumstances, and qualitative in others

• An enrollee survey question that asks
enrollees to rank their experience of
care on a scale from 1 (poor
experience) to 5 (excellent experience)

Nominal

The set of categories for a qualitative variable

• Mode of transportation to work (car,
bus, subway, bicycle, walk)

Data availability should be considered when selecting variables for PIPs, as more frequent access
to data, such as on a monthly, quarterly, or semi-annual basis, supports continuous quality
improvement (QI) and Plan Do Study Act (PDSA) efforts and can allow an MCP or state to
correct or revise course more quickly, if needed. If plans collect monthly, quarterly, or semiannual data, the plan should use a methodology to ensure comparability, such as a rolling 12month methodology. CMS encourages states to select PIP variables and performance measures
that can be examined on at least a semi-annual basis.
To the extent possible, CMS encourages MCPs to choose variables for PIPs that reflect health
outcomes. Performance measures are then used to measure these outcomes. For this protocol,
performance measures are used to monitor the performance of individual MCPs at a point in
time, to track MCP performance over time, to compare performance among MCPs, and to inform
the selection and evaluation of quality improvement activities. In addition, for the purpose of this
protocol, “outcomes” are defined as changes in patient health, functional status, satisfaction, or
goal achievement that result from health care or supportive services. 32 For example, measures of
avoidable hospitalizations or emergency department visits can demonstrate the adequacy of
access to preventive and primary care and effectiveness of care for acute and chronic conditions.
CMS recognizes that standardized performance measures addressing outcomes may be limited
because of the lag in observing changes in population health relative to the timeframe for the PIP
measurement period. Moreover, health outcomes may be influenced by factors outside of the
organization’s control, such as poverty, genetics, and environmental factors. For these reasons,
PIP outcomes do not always need to be health outcomes per se, but should be linked to health
outcomes.
Figure 1.2 provides guidance for selecting PIP performance measures for tracking performance
and improvement in outcomes over time. When selecting performance measures for a PIP, the

See 42 CFR 438.320.

32 | PROTOCOL ONE

MCP should first consider existing measures because the specifications for these measures often
have been refined over time, may reflect current clinical guidance, and may have benchmarks for
assessing MCP performance. CMS encourages use of the CMS Child and Adult Core Set, Core
Quality Measure Collaborative, and certified community behavioral health clinics (CCBHC)
measures. 33 Additional examples of existing measures include NCQA’s Healthcare Effectiveness
Data Information Set (HEDIS®) or measures that have been developed by the Agency of
Healthcare Research and Quality (AHRQ), such as the prevention quality indicators, inpatient
quality indicators, patient safety indicators, and pediatric quality indicators. 34
Figure 1.2. Guidance for Selecting PIP Performance Measures

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityMeasures/Core-Measures.html.More information about measures for behavioral health clinics
is available at https://www.samhsa.gov/section-223/quality-measures.

More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.

PROTOCOL ONE | 33

When there are gaps in existing measures, the MCP may develop new measures based on current
clinical practice guidelines or health services research. The MCP should consider the following
questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study question?

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the MCP to calculate the measure reliably and
accurately? Are there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

Step 6: Review the Data Collection Procedures
In this step, the EQRO assesses the validity and reliability of
the procedures the MCP used to collect the data that inform
the PIP measurements. Validity means that the data are
measuring what is intended to be measured. Reliability means
that the data are producing consistent results.
To ensure validity and reliability of the data collected as part
of the PIP, the MCP should develop a data collection plan that
specifies:
•

The data sources for the PIP

•

The data to be collected

•

How and when the data are to be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments used to collect the data

WORKSHEET 1.6
Resource for Activity 1, Step 6
Worksheet 1.6. Review the Data
Collection Procedures
• Provides a template for reviewing the
appropriateness of study variables
and performance measures to track
improvement
• Includes an assessment of data
collection procedures overall and for
administrative and medical record
review

This step may involve three main kinds of data collection: administrative data sources, medical
record review, and a hybrid method. The hybrid method uses a mix of both kinds of data
collection. Procedures to collect data from administrative data systems will be different from
procedures for visual inspection of medical records or other primary source documents.
However, both types of data collection require assurances that data are valid and reliable. CMS
encourages states to utilize data sources that they are able to collect data from on a regular basis
(e.g., monthly, quarterly, and semi-annually).
•

Administrative data collection. Evaluating an administrative data collection methodology
emphasizes the system that stores the data and should focus on an estimation of the degree of

34 | PROTOCOL ONE

completeness of the administrative data used to measure performance and track
improvement. See Section 2 of Worksheet 1.6 for a checklist of administrative data
assessment questions. In addition, refer to Protocol 5, Validation of Encounter Data Reported
by the Managed Care Plan for more information on assuring the validity and reliability of
encounter data.
•

Medical record review. For some variables, medical record review may be the only valid
and reliable source of data. (Note that medical records may include other sources besides the
individual patient medical record, such as clinical tracking logs, manual registries, case
management records, and the like.) If the PIP requires medical record reviews, special
attention should be given to the qualifications of the medical record reviewers, the specificity
of the guidelines for data collection, and plans for ensuring inter- and intra-rater reliability.
The reviewers should have a standard protocol for reviewing records, have the knowledge to
interpret the records, and have been trained to identify and code the information in the
records using consistent decision rules. See Section 3 of Worksheet 1.6 for a checklist of
medical record review assessment questions.

•

Hybrid data collection. The hybrid method uses both administrative and medical record
data. The hybrid method, when available, should be used when administrative data or
electronic health record (EHR) data are incomplete or may be of poor quality, or the data
elements for the measure are not captured in administrative data.

Step 7: Review Data Analysis and Interpretation of PIP Results
In this step, the EQRO assesses the quality of the data
analysis and interpretation of PIP results. The review
assesses whether the appropriate techniques were used,
and if the analysis and interpretation was accurate. In
addition, analysis and interpretation of the PIP data should
be based on a continuous quality improvement
philosophy 35 and reflect an understanding of lessons
learned and opportunities for improvement. Interpretation
of the PIP results should involve assessing the causes of
less-than-optimal performance and collecting data to
support the assessment.

WORKSHEET 1.7
Resource for Activity 1, Step 7
Worksheet 1.7. Review Data Analysis
and Interpretation of PIP Results
• Provides a template for assessing the
quality and completeness of the
analysis

Accurate data analysis is essential because the state or MCP may implement changes based on
the results. The primary source for the assessment should be analytic reports of PIP results
prepared by the MCP, including both baseline and repeat measurements of PIP outcomes. In

Continuous Quality Improvement (CQI) refers to the ongoing study of processes to (1) improve services or outcomes, and (2)
prevent or minimize the chance of adverse outcomes. To do so, the organization identifies areas for improvement and tests
approaches.

PROTOCOL ONE | 35

addition, the EQRO may assess the reasonableness of individual MCP results in relation to
existing state-level data, data from other MCPs, or industry benchmarks.
This protocol requires EQROs to assess the extent to which any change in performance is
statistically significant; however, it does not specify a level of statistical significance that must
be met. MCPs should indicate the level of statistical significance used in the analysis and which
findings were statistically significant.

Step 8: Assess the Improvement Strategies
In this step, the EQRO assesses the appropriateness of
WORKSHEET 1.8
the interventions for achieving improvement. This
assessment builds on the interpretation of PIP results in
Resource for Activity 1, Step 8
Step 7. Significant, sustained improvement results from
Worksheet 1.8. Assess the Improvement
developing and implementing effective improvement
Strategies
strategies (including strategies that are culturally and
• Provides a template for assessing
linguistically appropriate for the target population).
whether the selected improvement
Selected strategies should be evidence-based, that is,
strategies were appropriate for
achieving improvement
there should be existing evidence (published or
unpublished) suggesting that the test of change would
be likely to lead to the desired improvement in processes or outcomes (as measured by the
variables). Using the criteria in Worksheet 1.8, the EQRO should assess whether there is
evidence that the selected interventions were appropriate to achieve the aim of the PIP.
A common approach used to guide quality improvement work is the Institute for Healthcare
Improvement’s (IHI) Model for Improvement (see Box 1.4). After using this model to define the
parameters for the improvement effort, the MCP may test changes on a small scale using PDSA
cycles. PDSA cycles provide a methodology to test changes on a small scale and to apply rapidcycle learning principles to adjust intervention strategies over the course of the improvement.
This approach involves a continuous cycle of measuring and analyzing performance and requires
frequent review and adjustment. Data are evaluated on a regular basis and interventions are then
adapted based on what was learned. Interventions can then be scaled to larger settings or
populations if found effective. PIPs based on the Model for Improvement and PDSA process are
sometimes known as rapid-cycle PIPs. The EQRO can use results from PDSA cycles to inform
its assessment of the appropriateness of interventions to achieve the aim of the PIP.

36 | PROTOCOL ONE

Box 1.4. Using the IHI Model for Improvement to Assess Improvement Strategies
The IHI Model for Improvement provides a framework for conducting improvement work. The Model asks three
questions:
• What is your aim, and by when do you want to accomplish the aim?
• How will you know that a change is an improvement?
• What changes can you put in place to achieve your aim?
A PDSA cycle is used to structure the testing of improvement strategies. The steps in the PDSA cycle are to:
• Plan. Plan the test or observation, including a plan for collecting data, and interpreting results
• Do. Try out the test on a small scale
• Study. Set aside time to analyze the data and assess the results
• Act. Refine the change, based on what was learned from the test. Determine how to sustain the intervention, if
successful
This information about the Model for Improvement and PDSA approach was adapted from the Institute for Healthcare
Improvement, available at
http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx.
An additional source of information is the Agency for Healthcare Research and Quality’s Health Care Innovations
Exchange, available at https://innovations.ahrq.gov/qualitytools/plan-do-study-act-pdsa-cycle.

Step 9: Assess the Likelihood that Significant and Sustained Improvement
Occurred
In this step, the EQRO assesses the likelihood that
significant and sustained improvement occurred as a result
of the PIP. This assessment builds on findings from the two
previous steps. Box 1.5 suggests potential sources of
information for this assessment.

WORKSHEET 1.9
Resource for Activity 1, Step 9
Worksheet 1.9. Assess the Likelihood
that Significant and Sustained
Improvement Occurred

The EQRO should review the PIP methods and findings to
• Provides a template for assessing
assess whether there is evidence of statistically significant
improvement as a result of the PIP
improvement that may be associated with the intervention
implemented as part of the PIP. In addition, the EQRO may supplement the quantitative
assessment with information gathered through interviews with MCP staff and/or providers about
the implementation and results of the PIP intervention. Qualitative information may inform the
assessment of whether observed changes were likely to be attributable to the PIP intervention, as
opposed to a short-term event unrelated to the intervention or random chance.

PROTOCOL ONE | 37

Box 1.5. Potential Sources of Supporting Information
• Statistical significance testing calculated on baseline and repeat indicator measurements (clarify that the
appropriate test was used, such as a t-test for small samples)
• Benchmarks for quality specified by the state Medicaid agency or found in industry standards
• Interviews with MCP staff and providers about the implementation and results of the PIP intervention

An important goal of a PIP is to demonstrate sustained improvement. The EQRO should assess
whether repeated measurements were conducted, and if so, whether significant change in
performance relative to baseline measurement was observed. The repeat measurement should use
the same methodology as the baseline measurement. Any deviations in methodology (such as
sampling, data source, or variable definition) must be thoroughly documented. If the PIP is in the
early stages of implementation, and repeated measurements are not yet available, the analysis
plan should describe the methodology for subsequent measurement. The EQRO should state in
its final report which findings were found to be significant from a clinical and/or programmatic
perspective.

Activity 2: Perform Overall Validation and Reporting of PIP
Results
The EQRO should complete the two steps in Activity 2,
listed below, and answer the questions posed in each step.
In this activity, the EQRO assesses the overall validity and
reliability of the PIP methods and findings to determine
whether or not it has confidence in the results. The EQRO
will assign two validation ratings based on the EQRO’s
assessment of whether the PIP (1) adhered to acceptable
methodology for all phases of design and data collection,
conducted accurate data analysis and interpretation of PIP
results, and (2) produced evidence of improvement. The
ratings will be scored on a scale of high, moderate, low, or
no confidence.

WORKSHEET 1.10
WORKSHEET 1.11
Resources for Activity 2
Worksheet 1.10. Perform Overall
Validation of PIP Results
• Provides a template to provide a
validation rating (high confidence,
moderate confidence, low
confidence, or no confidence)
Worksheet 1.11. Framework for
Summarizing Information about PIPs
• Provides a structure for reporting

To assign the validation ratings, the EQRO will review the
general information about the PIP,
assessments conducted as part of the nine steps in Activity
performance measures and results,
validation results, and
1, and recorded in the Worksheets for Protocol 1: PIP
recommendations
Validation Tools and Reporting Framework, or a similar
tool. As studies always have weaknesses, the EQRO will
need to assess the relative strengths and weaknesses and the extent to which they affect the
confidence in the generalizability and usefulness of the PIP’s findings. CMS suggests using the
following validation rating to facilitate comparisons across PIPs and across states: high
confidence, moderate confidence, low confidence, and no confidence.
38 | PROTOCOL ONE

The EQRO will report its findings to the state and the state will submit the final technical report
to CMS. 36 The validation report should include a description of the PIPs that were validated and
the findings of the EQRO’s validation review. The EQRO and the state must include the actual
validation results of the PIPs in the final EQRO technical report for submission to CMS. The
EQRO is required to report the performance measurement data for the PIP validation in the EQR
technical report. 37 Please see “Tips for Drafting EQR Reports” in the Introduction for further
guidance to EQROs about how to produce a clear and concise report. In addition, please see
Worksheet 1.11. Framework for Reporting Summary PIP Information, for a suggested format for
summarizing PIP validation results in the EQR technical report.

Activity 3: Verify PIP Findings (Optional)
A state may request that the EQRO verify the data produced by the MCP to determine if the
baseline and repeated measurements are accurate. While the validation of the PIP methodology
and findings is a mandatory activity, the verification of data or performance measures used in the
PIP is optional for EQROs. Verification activities can provide added confidence in reported PIP
results as they provide greater evidence that the findings are accurate.
However, verification is a resource-intensive activity that may not be necessary. For example, if
the PIP uses HEDIS® measures that have been certified by a third party, verification may not be
needed. Additionally, the Information System Capabilities Assessment (ISCA) may provide
assurances that the processes used to develop measures for the PIP are valid and reliable (See
Appendix A. Information System Capabilities Assessment). Similarly, if the PIP relies on
encounter data and the EQRO has conducted encounter data validation, the optional EQR-related
activity described in Protocol 5, further assurances may be provided about the accuracy and
completeness of the data used in the PIP.
If a state opts to have the EQRO verify the accuracy of the baseline and repeated measurements,
the EQRO should focus on the processes through which data for the PIP were obtained,
processed, and analyzed. The verification process should begin with a thorough review of
existing resources:
•

Documentation produced by the MCP about the data, algorithms, and testing (e.g., code
reviews) related to the PIP data analysis

•

The assessment of the MCP’s information system produced as part of the ISCA

•

Any external validations of the accuracy and completeness of MCP encounter data (such as
the optional EQR-related activity)

For the purposes of the EQR protocols and ease of explanation, we refer to EQROs as the entity conducting the EQR-related
activities.

CHIP regulations at 42 CFR 457.1250 cross-reference to 42 CFR 438.364(a)(2)(iii).

PROTOCOL ONE | 39

•

Results of other EQR-related activities, such as performance measure validation or
compliance reviews

•

Results of private accreditation reviews or state Medicaid agency audits

In the event that no current assessment of an MCP’s information system or encounter data exists,
the state may choose to contract this function to assist in verifying the accuracy of the PIPs.
Next, the EQRO should review specific algorithms and results related to the PIP measures.
Questions include:
•

Was the algorithm used to produce the PIP measures sound (that is, does the algorithm
measure what it is intended to measure, are the results consistent, and is the code well
documented)?

•

For measures calculated using administrative data: Did the MCP’s information system
capture enrollee information completely and accurately? To answer this question, the EQRO
may need to validate a sample of records to ensure the encounter data are complete

•

For measures produced through medical record review: Did the MCP conduct a reabstraction of a small subset (validation sample) of the reviewed records to ensure the
abstraction was complete and accurate? Data retrieval and analysis should be conducted on a
small scale, with the validation sample following the same rules as the original PIP.

If validation of a sample of records is performed, the EQRO should perform statistical
correlations between the validation sample and the original PIP data. A variety of statistical
methods can be applied to assess the degree of correlation between the PIP and validation
measures. Two recommended methods are the Pearson correlation coefficient for continuous
data (e.g., age, income) and the Kappa statistic for categorical data (e.g., gender, race). Assessing
the algorithm together with the integrity of the data will provide a strong indication of the
accuracy of the PIP s findings.
END OF PROTOCOL 1

40 | PROTOCOL ONE

Worksheets for Protocol 1: PIP Validation Tools and
Reporting Framework
Instructions. Use these or similar worksheets to assist in validating PIPs conducted by the MCP. These worksheets
provide templates for validating PIPs and a framework for reporting on validated PIPs in the EQR technical report.
This tool includes the following worksheets crosswalked to the applicable Activity and Step:
Worksheet Name

Protocol Activity and Step

Worksheet 1.1. Review the PIP Topic

Activity 1. Step 1. Review the Selected PIP Topic

Worksheet 1.2. Review the PIP Aim Statement

Activity 1. Step. 2. Review the PIP Aim Statement

Worksheet 1.3. Review the Identified PIP Population

Activity 1. Step 3. Review the Identified PIP Population

Worksheet 1.4. Review the Sampling Method

Activity 1. Step 4. Review the Sampling Method

Worksheet 1.5. Review the Selected PIP Variables

Activity 1. Step 5. Review the Selected PIP Variables

Worksheet 1.6. Review the Data Collection Procedures

Activity 1. Step 6. Review the Data Collection
Procedures

Worksheet 1.7. Review Data Analysis and Interpretation
of PIP Results

Activity 1. Step 7. Review Data Analysis and
Interpretation of PIP Results

Worksheet 1.8. Assess the Improvement Strategies

Activity 1. Step 8. Assess the Improvement Strategies

Worksheet 1.9. Assess the Likelihood that Significant
and Sustained Improvement Occurred

Activity 1. Step 9. Assess the Likelihood that Significant
and Sustained Improvement Occurred

Worksheet 1.10. Perform Overall Validation of PIP
Results

Activity 2. Perform Overall Validation and Reporting of
PIP Results

Worksheet 1.11. Framework for Summarizing Information Activity 2. Perform Overall Validation and Reporting of
about Performance Improvement Projects (PIPs)
PIP Results
For each PIP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
EQRO interview date
Performance Improvement Project (PIP) name
PIP period date

MM/DD/YY to MM/DD/YY

Type of delivery system (check all that apply)

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM entity □ LTSS
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

Enrollees

Physicians

41 | PROTOCOL ONE 41

# Medicaid and CHIP enrollees in MCP: ____

# MCP primary care physicians: _____

# Medicaid and CHIP enrollees in the PIP: ____

# MCP specialty physicians: _____

# Total number of MCP enrollees in the PIP: ____
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

42 | PROTOCOL ONE

Worksheet 1.1. Review the Selected PIP Topic
PIP Topic ______________________________________________________________________________
Assess the appropriateness of the selected PIP topic by answering the following questions about the MCP and PIP.
Insert comments to explain “No” and “Not applicable (NA)” responses.
Question

Yes

Comments

1.1 Was the PIP topic selected through a
comprehensive analysis of MCP enrollee needs,
care, and services (e.g., consistent with
demographic characteristics and health risks,
prevalence of conditions, or the need for a specific
service by enrollees)? (If the PIP topic was required
by the state, please check “not applicable” and note
in comments.)
1.2 Did selection of the PIP topic consider
performance on the CMS Child and Adult Core Set
measures?
1.3 Did the selection of the PIP topic consider input
from enrollees or providers who are users of, or
concerned with, specific service areas? (If the PIP
topic was required by the state, please check “not
applicable” and note in comments.)
• To the extent feasible, input from enrollees who
are users of, or concerned with, specific services
areas should be obtained.
1.4 Did the PIP topic address care of special
populations or high priority services, such as:
• Children with special health care needs
• Adults with physical disabilities
• Children or adults with behavioral health issues
• People with intellectual and developmental
disabilities
• People with dual eligibility who use long-term
services and supports (LTSS)
• Preventive care
• Acute and chronic care
• High-volume or high-risk services
• Care received from specialized centers (e.g.,
burn, transplant, cardiac surgery)
• Continuity or coordination of care from multiple
providers and over multiple episodes
• Appeals and grievances
• Access to and availability of care
1.5 Did the PIP topic align with priority areas
identified by HHS and/or CMS?
1.6 Overall assessment: In the comments section,
note any recommendations for improving the PIP
topic.

PROTOCOL ONE | 43

Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

44 | PROTOCOL ONE

Worksheet 1.2. Review the PIP Aim Statement
PIP Aim Statement ______________________________
Assess the appropriateness of the selected PIP topic by answering the following questions. Insert comments to
explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

2.1 Did the PIP aim statement
clearly specify the improvement
strategy, population, and time
period for the PIP?
2.2 Did the PIP aim statement
clearly specify the population for
the PIP?
2.3 Did the PIP aim statement
clearly specify the time period
for the PIP?
2.4 Was the PIP aim statement
concise?
2.5 Was the PIP aim statement
answerable?
2.6 Was the PIP aim statement
measurable?
2.7 Overall assessment: In the
comments section, note any
recommendations for improving
the PIP aim statement.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL ONE | 45

Worksheet 1.3. Review the Identified PIP Population
PIP Population _______________________________
Assess whether the study population was clearly identified by answering the following questions. Insert comments to
explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

3.1 Was the project population clearly
defined in terms of the identified study
question (e.g., age, length of the study
population’s enrollment, diagnoses,
procedures, other characteristics)?
• The required length of time will vary
depending on the PIP topic and
performance measures
3.2 Was the entire MCP population
included in the PIP?
3.3 If the entire population was included in
the PIP, did the data collection approach
capture all enrollees to whom the PIP
question applied?
• If data can be collected and analyzed
through an administrative data system,
it may be possible to study the whole
population. For more guidance on
administrative data collection, see
Worksheet 1.6.
3.4 Was a sample used? (If yes, use
Worksheet 1.4 to review sampling
methods).
• If the data will be collected manually
(such as through medical record
review), sampling may be necessary
3.5 Overall assessment: In the comments
section, note any recommendations for
identifying the project population.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

46 | PROTOCOL ONE

Worksheet 1.4. Review the Sampling Method
Overview of Sampling Method _________________________________________________
If HEDIS® sampling is used, check here, and skip the rest of this worksheet.
Assess whether the sampling method was appropriate by answering the following questions. Insert comments to
explain “No” and “Not Applicable (NA)” responses. Refer to Appendix B for an overview of sampling approaches for
EQR data collection activities.
Question

Yes

Comments

4.1 Did the sampling frame contain a
complete, recent, and accurate list of
the target PIP population?
• A sampling frame is the list from
which the sample is drawn. It
includes the universe of members of
the target PIP population, such as
individuals, caregivers, households,
encounters, providers, or other
population units that are eligible to be
included in the PIP. The
completeness, recency, and accuracy
of the sampling frame are key to the
representativeness of the sample
4.2 Did the sampling method consider
and specify the true or estimated
frequency of the event, the confidence
interval to be used, and the acceptable
margin of error?
4.3 Did the sample contain a sufficient
number of enrollees taking into account
non-response?
4.4 Did the method assess the
representativeness of the sample
according to subgroups, such as those
defined by age, geographic location, or
health status?
4.5 Were valid sampling techniques
used to protect against bias? Specify
the type of sampling used in the
“comments” field.
4.6 Overall assessment: In the
comments section, note any
recommendations for improving the
sampling method.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL ONE | 47

Worksheet 1.5. Review the Selected PIP Variables and Performance Measures
Selected PIP Variables and Performance Measures:
_________________________________________________________________
Assess whether the selected PIP variables were appropriate for measuring performance and tracking improvement
by answering the following questions. Insert comments to explain “No” and “Not Applicable (NA)” responses.
Recall that CMS encourages MCPs to choose variables for PIPs that reflect health outcomes. Performance measures
are then used to measure these health outcomes. When selecting variables, the MCP should consider existing
performance measures.
Question
PIP variables
5.1 Were the variables adequate to answer the
PIP question?
• Did the PIP use objective, clearly defined, timespecific variables (e.g., an event or status that
can be measured)?
• Were the variables available to measure
performance and track improvement over time?
(CMS encourages states to select variables
that can be examined on at least a semi-annual
basis
Performance measures
5.2 Did the performance measure assess an
important aspect of care that will make a
difference to enrollees’ health or functional
status?
5.3 Were the performance measures appropriate
based on the availability of data and resources to
collect the data (administrative data, medical
records, or other sources)?
5.4 Were the measures based on current clinical
knowledge or health services research?
• Examples may include:
○ Recommended procedures
○ Appropriate utilization (hospital admissions,
emergency department visits)
○ Adverse incidents (such as death, avoidable
readmission)
○ Referral patterns
○ Authorization requests
○ Appropriate medication use

48 | PROTOCOL ONE

Yes

Comments

Question

Yes

Comments

5.5 Did the performance measures:
• Monitor the performance of MCPs at a point in
time?
• Track MCP performance over time?
• Compare performance among MCPs over
time?
• Inform the selection and evaluation of quality
improvement activities?
5.6 Did the MCP consider existing measures,
such as CMS Child and Adult Core Set, Core
Quality Measure Collaborative, certified
community behavioral health clinics (CCBHC)
measures, HEDIS®, or AHRQ measures?
5.7 If there were gaps in existing measures, did
the MCP consider the following when developing
new measures based on current clinical practice
guidelines or health services research?
• Did the measure address accepted clinical
guidelines relevant to the PIP question?
• Did the measure address an important aspect
of care or operations that was meaningful to
MCP enrollees?
• Did available data sources allow the MCP to
calculate the measure reliably and accurately?
• Were all criteria used in the measure defined
clearly (such as time periods, characteristics of
eligible enrollees, services to be assessed, and
exclusion criteria)?
5.8 Did the measures capture changes in enrollee
satisfaction or experience of care?
• Although enrollee satisfaction/experience is an
important outcome of care in clinical areas,
improvement in satisfaction should not be the
only measured outcome of a clinical project.
Some improvement in health or functional
status should also be addressed
• For projects in nonclinical areas (such as
addressing access or availability of services),
measurement of health or functional status is
preferred
5.9 Did the measures include a strategy to ensure
inter-rater reliability (if applicable)?

PROTOCOL ONE | 49

Question

Yes

Comments

5.9 If process measures were used, is there
strong clinical evidence indicating that the
process being measured is meaningfully
associated with outcomes?
• This determination should be based on
published guidelines, including citations from
randomized clinical trials, case control studies,
or cohort studies
• At a minimum, the PIP should be able to
demonstrate a consensus among relevant
practitioners with expertise in the defined area
who attest to the importance of a given process
5.10 Overall assessment: In the comments
section, note any recommendations for improving
the selected PIP variables and performance
measures.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

50 | PROTOCOL ONE

Worksheet 1.6. Review the Data Collection Procedures
Assess whether the data collection procedures were valid and reliable by answering the following questions. This
worksheet includes three sections: (1) overall data collection procedures, (2) data collection procedures for
administrative data sources, and (3) data collection procedures for medical record review. Insert comments to explain
“No” and “Not Applicable (NA)” responses.

Section 1: Assessment of Overall Data Collection Procedures
Question

Yes

Comments

6.1 Did the PIP design specify a systematic
method for collecting valid and reliable data that
represents the population in the PIP?
6.2 Did the PIP design specify the frequency of
data collection? If yes, what was the frequency
(for example, semi-annually)?
6.3 Did the PIP design clearly specify the data
sources?
• Data sources may include:
○ Encounter and claims systems
○ Medical records
○ Case management or electronic visit
verification systems
○ Tracking logs
○ Surveys
○ Provider and/or enrollee interviews
6.4 Did the PIP design clearly define the data
elements to be collected?
• Accurate measurement depends on clear and
concise definitions of data elements (including
numerical definitions and units of measure)
6.5 Did the data collection plan link to the data
analysis plan to ensure that appropriate data
would be available for the PIP?
6.6 Did the data collection instruments allow for
consistent and accurate data collection over the
time periods studied?
6.7 If qualitative data collection methods were
used (such as interviews or focus groups), were
the methods well-defined and designed to collect
meaningful and useful information from
respondents?
6.8 Overall assessment: In the comments section,
note any recommendations for improving the data
collection procedures.
Note: Include assessment of data collection
procedures for administrative data sources and
medical record review noted below.

PROTOCOL ONE | 51

Section 2: Assessment of Data Collection Procedures for Administrative Data Sources
Question

Yes

Comments

6.9 If inpatient data was used, did the data
system capture all inpatient
admissions/discharges?
6.10 If primary care data was used, did primary
care providers submit encounter or utilization
data for all encounters?
6.11 If specialty care data was used, did
specialty care providers submit encounter or
utilization data for all encounters?
6.12 If ancillary data was used, did ancillary
service providers submit encounter or utilization
data for all services provided?
6.13 If LTSS data was used, were all relevant
LTSS provider services included (for example,
through encounter data, case management
systems, or electronic visit verification (EVV)
systems)?
6.14 If EHR data was used, were patient,
clinical, service, or quality metrics validated for
accuracy and completeness as well as
comparability across systems?

Section 3: Assessment of Data Collection Procedures for Medical Record Review
Question
6.15 Was a list of data collection personnel
and their relevant qualifications provided?
• Data collection personnel require the
conceptual and organizational skills to
abstract data. These skills will vary
depending on the nature of the data and the
degree of professional judgment required.
For example, trained medical assistants or
medical records clerks may collect data if
the abstraction involves verifying the
presence of a diagnostic test report.
However, experienced clinical staff (such as
registered nurses) should be used to extract
data to support a judgment about whether
clinical criteria are met
6.16 For medical record review, was inter-rater
and intra-rater reliability described?
• The PIP should also consider and address
intra-rater reliability (i.e., reproducibility of
judgments by the same abstractor at a
different time)

52 | PROTOCOL ONE

Yes

Comments

Question

Yes

Comments

6.17 For medical record review, were
guidelines for obtaining and recording the data
developed?
• A glossary of terms for each project should
be developed before data collection begins
to ensure consistent interpretation among
and between data collection staff
• Data collection staff should have clear,
written instructions, including an overview of
the PIP, how to complete each section of
the form or instrument, and general
guidance on how to handle situations not
covered by the instructions. This is
particularly important when multiple
reviewers are collecting data
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL ONE | 53

Worksheet 1.7. Review Data Analysis and Interpretation of PIP Results
Assess whether the data analysis and interpretation was appropriate by answering the following questions. Insert
comments to explain “No” and “Not Applicable” responses.
Question

Yes

Comments

7.1 Was the analysis conducted in accordance
with the data analysis plan?
7.2 Did the analysis include baseline and
repeat measurements of project outcomes?
7.3 Did the analysis assess the statistical
significance of any differences between the
initial and repeat measurements?
7.4 Did the analysis account for factors that
may influence the comparability of initial and
repeat measurements?
7.5 Did the analysis account for factors that
may threaten the internal or external validity of
the findings?
7.6 Did the PIP compare the results across
multiple entities, such as different patient
subgroups, provider sites, or MCPs?
• Comparing the performance across multiple
entities involves greater statistical design
and analytical considerations than those
required for a project assessing
performance of a single entity, such as an
MCP, over time
7.7 Were PIP results and findings presented in
a concise and easily understood manner?
7.8 To foster continuous quality improvement,
did the analysis and interpretation of the PIP
data include lessons learned about less-thanoptimal performance?
• Analysis and interpretation of the PIP data
should be based on a continuous
improvement philosophy and reflect on
lessons learned and opportunities for
improvement
7.9 Overall assessment: In the comments
section, note any recommendations for
improving the analysis and interpretation of
PIP results
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

54 | PROTOCOL ONE

Worksheet 1.8. Assess the Improvement Strategies
Assess whether the selected improvement strategies were appropriate for achieving improvement by answering the
following questions. Insert comments to explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

8.1 Was the selected improvement strategy
evidence-based, that is, was there existing evidence
(published or unpublished) suggesting that the test of
change would be likely to lead to the desired
improvement in processes or outcomes (as
measured by the PIP variables)?
8.2 Was the strategy designed to address root
causes or barriers identified through data analysis
and quality improvement processes?
• Interventions that might have a short-term effect,
but that are unlikely to generate long-term change
(such as a one-time reminder letter to enrollees or
providers) are insufficient
• It is expected that interventions associated with
significant improvement will be system
interventions (such as educational efforts, policy
changes, or targeting of additional resources)
• It is expected that interventions should be
measurable on an ongoing basis (e.g., quarterly,
monthly) to monitor intervention progress
8.3 Was the rapid-cycle PDSA approach used to test
the selected improvement strategy?
• The steps in the PDSA cycle 38 are to:
○ Plan. Plan the test or observation, including a
plan for collecting data, and interpreting the
results
○ Do. Try out the test on a small scale
○ Study. Set aside time to analyze the data and
assess the results
○ Act. Refine the change, based on what was
learned from the test. Determine how to sustain
the intervention, if successful
• If tests of change were not successful (i.e., did not
achieve significant improvement), a process to
identify possible causes and implement solutions
should be identified
8.4 Was the strategy culturally and linguistically
appropriate? 39

Institute for Healthcare Improvement: Science of Improvement, Testing Changes. Available at
http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx.

More information on culturally and linguistically appropriate services may be found at
http://minorityhealth.hhs.gov/templates/browse.aspx?lvl=2&lvlID=15.

PROTOCOL ONE | 55

Question

Yes

Comments

8.5 Was the implementation of the strategy designed
to account or adjust for any major confounding
variables that could have an obvious impact on PIP
outcomes (e.g., patient risk factors, Medicaid
program changes, provider education, clinic policies
or practices)?
8.6 Building on the findings from the data analysis
and interpretation of PIP results (Step 7), did the PIP
assess the extent to which the improvement strategy
was successful and identify potential follow-up
activities?
8.7 Overall assessment: In the comments section,
note any recommendations for improving the
implementation strategies.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

56 | PROTOCOL ONE

Worksheet 1.9. Assess the Likelihood that Significant and Sustained Improvement
Occurred
Assess the likelihood that significant and sustained improvement occurred by answering the following questions.
Insert comments to explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

9.1 Was the same methodology used for baseline
and repeat measurements?
9.2 Was there any quantitative evidence of
improvement in processes or outcomes of care?
9.3 Was the reported improvement in performance
likely to be a result of the selected intervention?
• It is not necessary to demonstrate conclusively
(e.g., through controlled studies) that a change is
an effect of the intervention; it is sufficient to show
that the change might reasonably be expected to
result from the intervention
• It is not necessary to undertake data analysis to
correct for secular trends (e.g., changes that
reflect continuing growth or decline in a measure
because of external forces over an extended
period). The measured improvement should
reasonably be determined to have resulted from
the intervention
9.4 Is there statistical evidence (e.g., significance
tests) that any observed improvement is the result of
the intervention?
9.5 Was sustained improvement demonstrated
through repeated measurements over time?
9.6 Overall assessment: In the comments section,
note any recommendations for improving the
significance and sustainability of improvement as a
result of the PIP.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL ONE | 57

Worksheet 1.10. Perform Overall Validation of PIP Results
Provide two overall validation ratings of the PIP results. The first rating refers to the EQRO’s overall confidence that
the PIP adhered to acceptable methodology for all phases of design and data collection, and conducted accurate
data analysis and interpretation of PIP results. The second rating refers to the EQRO’s overall confidence that the
PIP produced evidence of significant improvement. Insert comments to explain the ratings. Provide comments to
justify the ratings.
PIP Validation Ratings (check one box)

Comments

Rating 1: EQRO’s Overall Confidence that the PIP
Adhered to Acceptable Methodology for All Phases
High confidence
Moderate confidence
Low confidence
No confidence
Rating 2: EQRO’s Overall Confidence that the PIP
Produced Evidence of Significant Improvement
High confidence
Moderate confidence
Low confidence
No confidence
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

58 | PROTOCOL ONE

Worksheet 1.11. Framework for Summarizing Information about Performance
Improvement Projects (PIPs)
To assist with the analysis portion of the EQR technical report requirement, Worksheet 1.11 should be completed in
its entirety for all PIPs. By doing so, it allows the EQRO to generate comparable information for all PIPs.

1. General PIP Information
Managed Care Plan (MCP) Name:
PIP Title:
PIP Aim Statement:
Was the PIP state-mandated, collaborative, statewide, or plan choice? (check all that apply)
State-mandated (state required plans to conduct a PIP on this specific topic)
Collaborative (plans worked together during the planning or implementation phases)
Statewide (the PIP was conducted by all MCOs and/or PIHPs within the state)
Plan choice (state allowed the plan to identify the PIP topic)
Target age group (check one):
Children only (ages 0–17)*

Adults only (age 18 and over)

Both adults and children

*If PIP uses different age threshold for children, specify age range here:
Target population description, such as duals, LTSS or pregnant women (please specify):
Programs:

Medicaid (Title XIX) only

CHIP (Title XXI) only

Medicaid and CHIP

2. Improvement Strategies or Interventions (Changes tested in the PIP)
Member-focused interventions (member interventions are those aimed at changing member practices or behaviors,
such as financial or non-financial incentives, education, and outreach)

Provider-focused interventions (provider interventions are those aimed at changing provider practices or behaviors,
such as financial or non-financial incentives, education, and outreach)

MCP-focused interventions/system changes (MCP/system change interventions are aimed at changing MCP
operations; they may include new programs, practices, or infrastructure, such as new patient registries or data
tools)

PROTOCOL ONE | 59

3. Performance Measures and Results (Add rows as necessary)
Performance
measures (be
specific and
indicate
measure
steward and
NQF number
if applicable):

Baseline
year

Baseline
sample
size and
rate

Most recent
remeasurement
year
(if applicable)

Most recent
remeasurement
sample size and
rate
(if applicable)

Demonstrated
performance
improvement
(Yes/No)

Statistically
significant change
in performance
(Yes/No)
Specify P-value

Yes
Not
applicable—PIP is
in planning or
implementation
phase, results not
available

Yes

Specify P-value:
<.01

<.05

Other (specify):

Yes
Not
applicable—PIP is
in planning or
implementation
phase, results not
available

Yes

Specify P-value:
<.01

<.05

Other (specify):

Yes
Not
applicable—PIP is
in planning or
implementation
phase, results not
available

Yes

Specify P-value:
<.01

<.05

Other (specify):

4. PIP Validation Information
Was the PIP validated?

Yes

“Validated” means that the EQRO reviewed all relevant part of each PIP and made a determination as to its validity. In many
cases, this will involve calculating a score for each relevant stage of the PIP and providing feedback and recommendations.

Validation phase (check all that apply):
PIP submitted for approval
First remeasurement

Planning phase

Second remeasurement

Implementation phase

Baseline year

Other (specify):

Validation rating #1: EQRO’s overall confidence that the PIP adhered to acceptable methodology for all phases of
design and data collection, conducted accurate data analysis and interpretation of PIP results,
High confidence

Moderate confidence

Low confidence

No confidence

Validation rating #2: EQRO’s overall confidence that the PIP produced significant evidence of improvement.
High confidence

Moderate confidence

EQRO comments on validation ratings

60 | PROTOCOL ONE

Low confidence

No confidence

EQRO recommendations for improvement of PIP:

END OF WORKSHEETS FOR PROTOCOL 1

PROTOCOL ONE | 61

Protocol 2. Validation of Performance
Measures
A MANDATORY EQR-RELATED ACTIVITY

ACTIVITY 2: CONDUCT SITE VISIT ACTIVITIES
ACTIVITY 3: CONDUCT POST-SITE VISIT ACTIVITIES

Background
States use performance measures to monitor the performance of individual
managed care plans (MCPs) at a point in time, to track performance over time,
to compare performance among MCPs, and to inform the selection and
evaluation of quality improvement activities. States specify standard
performance measures which the MCPs must include in their quality assessment
and performance improvement (QAPI) program. 40
In many cases, states and MCPs use measures included in the CMS Child and
Adult Core Sets to monitor and track quality of care in Medicaid and CHIP. 41
While use of these measures by states is voluntary, CMS encourages states to
adopt and use the Child and Adult Core Set measures to support their managed
care quality measurement and improvement initiatives. Many Core Set measures
are part of Healthcare Effectiveness Data and Information Set (HEDIS®), and
have national and regional benchmarks.
Federal regulations at 42 CFR 438.330(c) require states to specify standard
performance measures for MCPs to include in their comprehensive quality
assessment and performance improvement (QAPI) programs. 42 Each year, the
MCPs must: (1) measure and report to the state the standard performance
measures specified by the state; (2) submit specified data to the state which

More information about QAPI and performance measure validation is available at 42. CFR
438.330(b)(2) and (c), cross referenced by CHIP at 457.1240(b).

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/qualityof-care/performance-measurement/child-core-set/index.html. More information about the Adult Core
Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adultcore-set/index.html.

More information about QAPI and performance measures is available at 42. CFR 438.330(b)(2). This is
cross-referenced by CHIP at 42 C.RF.R 457.1240(b).

PROTOCOL TWO

ACTIVITY 1: CONDUCT PRE-SITE VISIT ACTIVITIES

enables the state to calculate the standard performance measures; or (3) a combination of these
approaches.
This protocol is used to guide the validation of the performance measures specified by states for
inclusion in MCPs’ QAPI programs. It applies both when the QAPI performance measure is
calculated by the MCP and when it is calculated by the state. In general, the external quality
review organization (EQRO) must assess whether the performance measures calculated by the
MCP 43 are accurate based on the measure specifications and state reporting requirements (42
CFR 438.330(b)(2)). The state provides the list of performance measures to be validated, the
specifications for the measures, and the requirements for reporting. As noted in the Introduction,
states have the option to use information from a Medicare or private accreditation review of an
MCP to provide information for the annual EQR instead of conducting this mandatory EQRrelated activity. 44, 45 A related protocol, Protocol 7. Calculation of Additional Performance
Measures, may be used by EQROs to calculate additional performance measures in accordance
with state specifications.

Getting Started on Protocol 2
Protocol 2 consists of three phases of activities: pre-site visit, site visit, and post-site visit (Figure
2.1). The activities take place before, during, and after the EQRO conducts a site visit with the
MCP. 46 The validation process is interactive and concurrent with MCP performance measure
calculation.

While the protocol is written as if the MCP is calculating performance measures, the MCP may contract with another entity to
calculate and report on its behalf. Alternatively, 42. CFR 438.330(c)(ii) allows the state to require the MCP to submit data to
the state, which the state then uses to calculate the performance measure. CHIP regulations cross-reference to these regulations
at 457.1240(b),. This protocol applies in either circumstance.

If the state elects to use nonduplication for this mandatory EQR-related activity (42 CFR 438.360 (Nonduplication of mandatory
activities with Medicare or accreditation review), then the state must ensure that all information from the Medicare or private
accreditation review is provided to the EQRO for analysis and inclusion in the annual EQR technical report. (See 42 CFR
438.360(a)(1)–(3) for additional details regarding the circumstance under which nonduplication is an option). Use of nonduplication
must be identified in the state’s quality strategy (see 42 CFR 438.360(c) and 438.340(b)(10)). CHIP cross-references to this
requirement at 42 CFR 457.1250, but does not allow for the use of Medicare review activities for the purposes of nonduplication.

A state may not utilize nonduplication if Medicare has accepted an only attestation of a plan’s QIP. In the context of this EQRrelated activity, the QIP would have to undergo validation as part of a Medicare review in order for nonduplication to be an
option. See 42 CFR 438.360(a)(2).

In the event that onsite activities are not feasible due to the COVID-19 pandemic, site visits may be conducted virtually.

PROTOCOL TWO | 63

Figure 2.1. Protocol 2 Activities

Two supplemental resources are available to help EQROs validate performance measures:
•

Worksheets for Protocol 2. Performance Measurement Validation Tools, which can be used
to prepare for and conduct pre-site, site, and post-site activities

•

Appendix A. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems

The remainder of this protocol outlines the steps associated with Activities 1 through 3.

64 | PROTOCOL TWO

Activity 1: Conduct Pre-Site Visit Activities
Step 1: Define the Scope of the Validation
The performance measures each state requires will depend
on the specific needs of the state. The state will provide the
EQRO with a list of the performance measures to be
validated along with requirements for data collection and
reporting (e.g., sampling guidelines and instructions for
calculating numerators and denominators).
The EQRO should use Worksheet 2.1. List of Measures to be
Validated to enumerate the performance measures to be
validated under Protocol 2, including their data source,
reporting frequency, and format. Five data sources are used
to produce MCP performance measures:
1. Administrative data, such as claims/encounter data,
registries, or vital records
2. Medical record review
3. Administrative data supplemented by medical record
review, referred to as the “hybrid” method
4. Electronic health records
5. Surveys (survey administration and validation is
addressed in Protocol 6)

WORKSHEET 2.1
WORKSHEET 2.2
Resources for Activity 1, Step 1
Worksheet 2.1. List of Performance
Measures to be Validated
• Provides a template for identifying the
measures the EQRO will validate for
the state, including the source, how
frequently to calculate each measure,
and when each measure is due to the
state
Worksheet 2.2. Performance Measure
Validation Template
• Provides a template for documenting
audit specifications for the validation
components of each performance
measure listed in Worksheet 2.1.,
and to assess the MCP’s
measurement and reporting process
for each component

For each of the measures to be validated, the EQRO should complete Worksheet 2.2.
Performance Measure Validation Template (or a similar tool). The worksheet is used to
systematically gather information about the validation components and audit specifications based
on existing documentation about the measure. Elements include:
•

Documentation related to the data collection and calculation method

•

Denominator calculation(s), including adequacy of the data sources to calculate the
denominator, operationalization of the measure-specific eligibility criteria, and adherence to
the measurement period

•

Numerator calculation(s), including adequacy of the data sources to calculate the numerator,
appropriateness of codes used to identify numerator compliance, avoidance of double
counting, and adherence to the measurement period

•

Sampling methodology (if used)

PROTOCOL TWO | 65

•

Reporting of rates and other supporting information, including documentation of deviations
(if any)

Worksheet 2.2 also contains an example of a completed performance measure validation
worksheet similar to what an EQRO would use before, during, and after its site visit. The
illustrative template is for Chlamydia Screening in Women (CHL-CH, Measure Steward:
National Committee for Quality Assurance, National Quality Forum [NQF] # 0033), which is
included in both the Child and Adult Core Sets and calculated using the administrative method.
During Activity 1, Step 1, the EQRO should begin to populate the audit specifications based on
the available measure documentation. Note that the worksheet is intended to serve as a “living
document” for the measure validation process and the EQRO can adapt the template if necessary.

Step 2: Assess the Integrity of the MCP’s Information System
This step helps focus the validation activities on aspects of
the MCP’s information system that are most likely to be an
issue in the validation process. Before validating individual
performance measures, the EQRO must assess (1) the
integrity of the MCP’s information system, (2) the
completeness and accuracy of the data produced, and (3) the
readiness of the MCPs’ data systems for calculating
performance measures. As part of this step, the EQRO
conducts an Information Systems Capabilities Assessment
(ISCA) for each MCP as described in the following
sections.

WORKSHEET A.1
WORKSHEET A.2
Resources to Conduct an
Information Systems
Capabilities Assessment (ISCA)
The ISCA is used to validate MCP
information systems, processes, and
data. The ISCA provides a foundation
for the validation of performance
measures.

Conduct an Information Systems Capabilities
Assessment

Appendix A

Before conducting the site visit, the EQRO should provide
the MCP with information on the ISCA process, including
Worksheet A.1 Information System Capabilities
Assessment Tool. The ISCA is used to validate MCP
information systems, processes, and data. The ISCA
corresponds to the objectives identified in this protocol and
addresses key components of calculating performance
measures including:

Worksheet A.1. ISCA Tool

•

General information about the MCP

•

Membership/enrollment data systems

•

Claims/encounter data processing

•

Provider data

66 | PROTOCOL TWO

• Explains how to conduct the ISCA

• Provides a template for MCPs to
document the capabilities of the
information systems, processes, and
data
Worksheet A.2. ISCA Interview Guide
• Provides a guide to EQROs for
conducting follow-up interviews with
MCP staff to record responses and
document specific issues based on
findings from Worksheet A.1

•

Data completeness

•

Integration of data for performance measure calculation

The ISCA provides information about the timing of any other recent, independent, documented
assessment such as a HEDIS Compliance Audit™. If the MCP recently had a comprehensive,
independent assessment of its information systems, the EQRO may review those results. If the
MCP has not had an ISCA within a timeframe determined by the state, 47 the EQRO will conduct
an ISCA as part of this protocol. It is recommended that EQROs request that MCPs provide any
assessments of their information technology (IT) systems conducted in the previous two years.
The EQRO should document the strengths and weaknesses of the MCP information system
relevant to the types of data used by the MCP in calculating performance measures. The EQRO
should take into account systems issues (such as missing data), when validating individual
performance measures and determining whether they are reportable.
Assess MCP Data Systems and Types
The EQRO should assess every data system and type of data the MCP processes to ensure the
required data are current and accurate, particularly at the time it extracts data for its performance
measures. The EQRO should assess changes in the MCP’s data systems that might affect the
production of the performance measures. Major changes, upgrades, or consolidations within the
system, or acquisitions/mergers with other MCPs may impact the accuracy or completeness of
required data elements. Elements that should be assessed for each MCP data system and type
include:
•

Membership/enrollment data

•

Provider data

•

Claims and encounter data

•

Medical records data

•

Pharmacy, laboratory, and other ancillary data

Membership/Enrollment Data
The EQRO should assess:
•

The MCP’s ability to track members over time, changes in enrollment, name changes, and
changes in coverage

There is no statutory or regulatory requirement for the frequency with which ISCAs should be conducted. Each state must
determine the maximum interval between assessments of MCP information systems, balancing the cost to the state and burden
on the MCP with the need to ensure that changes to the MCP’s information systems are assessed frequently enough to support
accurate performance measurement.

PROTOCOL TWO | 67

•

The MCP’s processes to ensure membership/enrollment data are current and accurate

•

Changes in the MCP’s membership data systems that might affect the production of the
performance measures

•

Whether transactions between the MCP and state data systems (such as state eligibility files)
affect measure calculation through updating, correcting, or overwriting source data (e.g., race
or ethnicity information)

The EQRO should determine whether each MCP member is uniquely identifiable and can be
linked to the state’s Medicaid and CHIP eligibility file. The membership/enrollment database
should capture the following information for every member:
•

Unique member identifier (ID), including state-issued Medicaid and CHIP ID and CMSissued Medicare number (if applicable)

•

Eligibility category

•

Date of birth

•

Sex

•

Race and ethnicity

•

Primary language

•

Disability status

•

Enrollment and/or termination dates, including multiple enrollment and termination dates
within and across programs (preferably exact dates rather than monthly indicators)

•

Primary care provider (e.g., provider name, provider ID number, provider location)

Collecting and assessing membership and enrollment data is increasingly important due to the
quality strategy requirement for identifying, evaluating, and reducing health disparities based on
age, race, ethnicity, sex, primary language, and disability status (42 CFR 438.340(b)(6)). Under
this requirement, states must identify this demographic information for each Medicaid enrollee
and provide it to the MCP, PIHP, or PAHP at the time of enrollment.
In addition, to facilitate geographic stratification of performance (such as analyses of access and
timeliness of care), complete and accurate information on the household’s location of residence
(e.g., ZIP code) is also desirable.
Finally, to facilitate surveys of patient experience, complete contact information is essential. At a
minimum, name and address are required; phone numbers and email addresses are highly
desirable. See Protocol 6. Administration or Validation of Quality of Care Surveys for more
information.

68 | PROTOCOL TWO

Provider Data
The EQRO should review an MCP’s provider data system(s) to assess the MCP’s ability to track
providers over time, across multiple office locations, and through changes in participation. In
addition, the EQRO should assess how many contracted providers use electronic health records
(EHRs) and the extent to which EHRs are used in the calculation of an MCP’s performance
measures.
Claims and Encounter Data
Claims and encounter data should cover all types of services offered by the MCP and not
separately contracted by the state, such as hospital inpatient, hospital outpatient, primary care,
skilled nursing facility, nursing facility (custodial care), specialty care, behavioral health care,
family planning services, home health care, radiology, laboratory, pharmacy, dental care, and
vision care. The EQRO should note the following for each type of claim/encounter data
captured:
•

Total number of diagnosis and procedure codes (such as Healthcare Common Procedure
Coding System (HCPCS) codes and Current Procedural Terminology (CPT)® codes, the
American Dental Association’s Common Dental Terminology (CDT)© codes, and ICD-10
Procedure Coding System codes), captured by the system

•

Whether the principal diagnosis, secondary diagnoses, and procedure codes can be accurately
distinguished in the system

•

Maximum number of digits/characters captured for each data field in each type of claim or
encounter

The accuracy and validity of measures may be adversely affected if the information system
truncates codes or is unable to collect and/or differentiate among a sufficient number of codes.
The EQRO should understand the various coding systems and forms used by the MCP and its
vendors to capture and process clinical information through its claims and encounter databases.
The EQRO should assess how well the information system translates or maps these codes back
to the criteria for MCP performance measure reporting, and how it ensures the accuracy of these
translation processes.
The EQRO should also determine, through review of existing documentation or in consultation
with the MCP, whether certain diagnosis or procedure codes required for performance
measurement are not accurately or completely captured in the claims and encounter data systems,
such as maternity or dental care, behavioral health care, and preventive care services.
Medical Record Data
The EQRO should use medical record data to review:
•

Methods used to retrieve information from medical records
PROTOCOL TWO | 69

•

Training and tools that medical record review staff receive

•

Processes used to ensure accurate data retrieval, inter-rater reliability, and data entry into a
database used to produce performance measures

With increasing adoption of EHRs and state use of Health Information Exchanges, MCPs and
provider practices may use newer methods to extract information from the medical record. As
noted earlier, the EQRO should assess how electronic records are used in performance measure
calculation, and whether there are any special considerations in the validity and reliability of
these records for accurate measurement.
Pharmacy, Laboratory, and Other Ancillary Data
Pharmacy data use standardized codes for prescription drugs such as those promulgated by the
National Council for Prescription Drug Programs (NCPDP). 48 Laboratory services frequently use
a similar, nationally recognized system of coding (known as LOINC®). 49
Due to the diversity in the size, type, and ownership of pharmacy, laboratory, and other ancillary
providers, non-standard codes should be examined. When found, the EQRO should assess the
MCP’s system for cross-walking these different codes to store the necessary information in its
performance measure database. The EQRO should understand the MCP’s mapping system of
non-standard codes to standardized codes and the mechanism used to ensure the accuracy of
these translation processes.
If the MCP does not collect pharmacy, laboratory, or other ancillary data through an
administrative or claims database, it may retrieve these data from medical records. However,
medical records often are an unreliable source due to non-standard coding and terminology, poor
coordination of records, and insufficient record linkages between primary care and specialist
providers. These issues should be addressed during the claims/encounter data review and the
medical record review, and, if necessary, reflected on any corrective action plan.
The EQRO must assess the ability of the information system to link these different sources of
data. For example, to identify enrollees with diabetes, a MCP may need to combine diagnosis
code data from inpatient or ambulatory encounters (not all ongoing conditions are reported at
every encounter) with pharmacy data, lab data, and/or a disease registry, an MCP’s disease
management system, or a medical management system used by MCP staff, if one exists. Thus, to
determine whether enrollees with diabetes have received a retinal examination from an
ophthalmologist or optometrist within the previous year, the MCP would have to link diagnosis
and procedure code data from encounter forms, medical records, and/or claims data with

More information about NCPDP codes is available at https://www.ncpdp.org/.

More information about LOINC codes is available at https://loinc.org/.

70 | PROTOCOL TWO

information about the specialty of the providers that performed the examinations for these
members.
Synthesis of Findings
The EQRO will review the findings from the ISCA across each of the data systems and types.
The EQRO should note any problem areas related to the adequacy of the MCP’s data systems to
calculate and report the required performance measures. Where a response is incomplete,
indicates an inadequate process, or requires clarification, the EQRO should flag the issue for
follow-up and further review during the site visit.

Step 3: Conduct Detailed Review of Measures
The next step is to conduct a detailed review of
measures, incorporating findings from the ISCA. In its
detailed review, the EQRO should identify measures that
are most vulnerable to inaccurate results based on its
knowledge of the MCP’s data systems and processes.
For example, if the MCP uses global billing for
maternity care, calculation of maternity measures could
be affected by the lack of separate claims for prenatal
and postpartum care, and thus, performance
measurement results for such measures could be
significantly under-reported. Similarly, the EQRO
should identify certain types of claims that may require
linkage from other data sources (such as laboratory,
behavioral health, or dental) because the necessary codes
may not be available for all members.
The detailed review of each measure involves a
systematic assessment of the code and output to assess
adherence to the specifications as well as the impact of
any systems issues on the accuracy and completeness of
the data. In addition, the EQRO should pay special
attention to frequently-encountered issues in developing
its audit specifications based on findings from the ISCA:

WORKSHEET 2.2
Resources for Detailed Review
of Measures
A detailed review of each measure
includes the following:
• Source code for the measure
• Data mapping, if applicable
• Measure workflow
• Data output at each stage of the
measure calculation
• Record-level numerator and
denominator data
Worksheet 2.2. Performance Measure
Validation Template
• Provide a template for documenting
audit specifications for the validation
components of each performance
measure listed in Worksheet 2.1.,
and to assess the MCP’s
measurement and reporting process
for each component

•

Claims-dependent denominators

•

Complex continuous enrollment criteria

•

Use of global billing

•

Identification of live births (including linkage of mother and infant records)

PROTOCOL TWO | 71

•

Procedure codes that are infrequently billed by providers (such as developmental screening,
documentation of Body Mass Index [BMI], or BMI percentile in the medical record)

•

Ability to link claims and pharmacy data

•

Identification of practitioner type (especially mental health providers)

•

Multiple numerator events

•

Vendor-supplied data

During the detailed measure review, the EQRO should develop targeted audit specifications for
each measure to account for potential systems issues. The EQRO should record its audit
specifications and interim findings on Worksheet 2.2. Performance Measure Validation
Template, or a similar worksheet. Box 2.1 provides additional information on validation of
HEDIS® measures calculated with HEDIS®-certified software.
Box 2.1. Review of HEDIS® Measures Calculated with HEDIS®-certified Software
• If the state requires HEDIS® measures and the MCP used HEDIS®-certified software to calculate the measures,
the EQRO does not need to review source code for those measures. However, the EQRO is required to verify that
the measures were calculated as specified by the software and that systems issues did not compromise the
accuracy and completeness of the performance measures. As an example, when an MCP pays for prenatal and
postpartum care as part of a bundled maternity care payment, HEDIS® measures may be calculated according to
the specifications but the rates may be significantly under-reported using administrative data due to the lack of
separate claims for prenatal and postpartum care. Thus, the EQRO is required to review and validate the accuracy
and completeness of HEDIS® measures based on findings from the ISCA.

Step 4: Initiate Review of Medical Record Data Collection
The purpose of this step is to verify the accuracy of the
medical record review conducted by the MCP when medical
record data are used to calculate and report performance
measures. If a plan only used administrative data, this step is
not necessary.
To validate the integrity of the medical record review
processes, the EQRO conducts the validation in two phases:
the first phase assesses the initial implementation of the
process to allow corrections at an early stage; the second
phase is a retrospective review of the accuracy of the
medical record review abstraction process.

WORKSHEET 2.3
Resource for Activity 1, Step 4
Worksheet 2.3. Medical Review
Validation Template
• Provides instructions for conducting
the medical record review and
worksheets to summarize reabstraction findings from the review
(Worksheet 2.3, Table 1) and to
record the impact of findings from the
review (Worksheet 2.3, Table 2)

Review of Implementation of Medical Record Review
During the early implementation of the medical record abstraction process, the EQRO will
confirm the following about MCP activities:
72 | PROTOCOL TWO

1. Selection of staff with appropriate experience and credentials
2. Development of high-quality abstraction tools to collect the required information
3. Provision of effective staff training about the review process
4. Implementation of sound oversight procedures to assess reviewer performance (such as
validation of a sample of records or tests of inter-rater reliability)
The EQRO may review a convenience sample of records across measures to identify potential
problems for MCP correction. NCQA’s HEDIS Compliance Audit™ recommends selecting up
to 10 difficult-to-review measures and obtain copies of at least 2 complete medical record review
tools and charts per measure. If the state requires fewer than 10 measures that rely on medical
record data, the EQRO should conduct the sample review for all medical record-dependent
measures. Completing this step early in the process allows the MCP to address identified issues
and resolve them during the initial stages of data collection.
Re-abstraction and Validation of Medical Record Review
The EQRO will conduct a retrospective medical record review for at least two measures that
include medical record review either alone or in combination with administrative data (known as
the hybrid method). The EQRO should target statistical validation to measures that are new,
complex, and dependent on the medical record data or those with previously identified issues.
For each measure, the EQRO will request a sample of 30 medical records with positive
numerator events and compare the completed abstraction information to the medical record to
determine the rate of agreement. If the agreement rate is less than 100 percent, the EQRO will
assess the degree of bias. Worksheet 2.3. Medical Record Review Validation Template provides
a detailed description of the medical record review process and validation tool. The EQRO
should summarize findings for the MCP from the medical record review validation, including
error rates for the measures that were validated (see Table 2 in Worksheet 2.3) and
recommendations for improving the medical record review process.

Step 5: Prepare for the MCP Site Visit
Before conducting the site visit, the EQRO will contact the
MCP to:
•

Explain the procedures and timeline for performance
measure validation activities

•

Communicate the EQRO’s policies and procedures for
safeguarding confidential information and signed
confidentiality agreements

•

Organize the site visit to ensure the availability of
necessary documentation and staff (see Box 2.2).

WORKSHEET 2.4
Resource for Activity 1, Step 5
Worksheet 2.4. Potential Documents
and Process for Review
• Provides a checklist of documents,
data, and procedures the MCP
should make available before or
during the site visit

PROTOCOL TWO | 73

Box 2.2. Potential Site Visit Participants
During the site visit, the MCP should arrange for staff and vendors to meet with the EQRO to provide information
about the processes to processes to calculate or report performance measures. The EQRO may want to suggest to
the MCP that corporate staff—particularly Information Systems (IS) staff—be included in the site visit as corporate
staff may provide additional insight into some interview questions. Participants may include:
• The Director of Health/Medical Information Systems
• Information system programmers or operators
• Director of Member/Patient Services and staff
• Director of Utilization Management and staff
• Director of Quality Improvement and staff

At this stage, the EQRO should also request confirmation of the list and description of staterequired performance measures. The EQRO will provide the MCP a list of documents, data, and
procedures that may be reviewed before or during the site visit (refer to Worksheet 2.4. Potential
Documents and Processes for Review).

Activity 2: Conduct Site Visit Activities
Site visit activities provide an opportunity for the EQRO to follow up on findings from the presite information system assessment and to confirm or clarify information about the production
and reporting of performance measures through document review or discussions (see Box 2.3).
Box 2.3. Purpose of the Site Visit
• Confirm, observe, and query systems used to produce performance measure results, including membership,
medical, pharmacy, provider, and other ancillary or supplemental data sources
• Investigate and follow up on issues identified from the ISCA
• Assess data integration and control procedures for accurate production of the performance measures
• Assess data completeness
• Confirm processes for calculating and reporting the performance measures

During the site visit, the EQRO will complete the following steps, which are described below:
1. Review the information systems underlying performance measurement
2. Assess data integration and control for performance measure calculation
3. Review performance measurement production
4. Complete the detailed review of measures
5. Assess the sampling process
6. Communicate preliminary findings and outstanding items

74 | PROTOCOL TWO

Step 1: Review Information Systems Underlying Performance Measurement
The review of the ISCA which had begun during the presite phase continues during the site visit. During this
phase, the EQRO reviews the information system
components that the MCP uses to produce performance
measures via (1) staff interviews, (2) primary source
documents, (3) systems and processes used to calculate
performance measures, (4) data entry observation, and (5)
data files. These sources are described below.
1.

Staff Interviews

WORKSHEET 2.5
Resource for Activity 2, Step 1
Worksheet 2.5. Interview Guide for MCP
Data Integration and Control Personnel
• Provides a list of interview questions
for key staff involved in the
production of performance measures
using questions tailored to the MCP’s
processes for producing these
measures. Tailor the questions as
appropriate

The EQRO will interview key staff (scheduled and
confirmed ahead of the visit) involved in the production of
performance measures using questions tailored to the MCP’s processes for producing
performance measures based on findings from the ISCA. These interviews also provide an
opportunity to supplement the review of information system policies, procedures, and data
(described below). See Worksheet 2.5. Interview Guide for MCP Data Integration and Control
Personnel.
2.

Primary Source Verification

The EQRO will review the primary source documents, including paper forms and other input to
the MCP systems, and confirm that the information from the primary source matches the
information used for performance measurement. In addition, the EQRO will review the processes
used to input, confirm entry, and identify errors, as well as processes used to transmit and track
the data through systems. Typical forms the EQRO will review include:
•

Member-initiated enrollment data

•

Hospital claims/encounters

•

Ambulatory claims/encounters

•

Prescription data

•

Practitioner demographic forms

•

Practitioner credentialing forms

•

Claims logs

•

Lab results

PROTOCOL TWO | 75

System and Process Review

The EQRO will review the MCP’s documentation describing the systems and processes used to
calculate performance measures to confirm they adhere to state policies and procedures. These
include systems and processes for collecting, storing, and reporting data. All documentation
received and examined must be recorded.
4.

Observation

The EQRO will observe key MCP processes required for performance measure calculation to
assess data entry and other data manipulations. Examples include:
•

Data entry of membership updates, claims/encounter data, and practitioner data (e.g., confirm
that mandatory fields are required and invalid data elements are identified, such as invalid
birth dates or invalid service dates)

•

Claims operations including overrides or exceptions

•

Computer operations and security plans to confirm procedures are followed

The EQRO will directly observe the Extract, Transform, and Load (ETL) process 50 and its
replication by two separate operators through the process using an observation guide to confirm
the activities, as well as the process where data are incomplete (e.g., a claim without a provider
identification number).
5.

Data File Review

The EQRO will directly examine data files to confirm the data are stored and processed
according to the documentation provided. Examples of files to review include:
•

Transaction files for clinical services, membership, and practitioner changes

•

Intermediate files created by extracts, queries, and analysis applications

•

Data repository files

ETL is when these three database functions (extract, transform, and load) are combined into one tool to pull data out of one
database and place it into another database.

76 | PROTOCOL TWO

Step 2: Assess Data Integration and Control for Performance Measure
Calculation
In this step, the EQRO will assess the MCP’s ability to link
data from multiple sources and the extent to which the MCP
has created systems and processes to ensure the accuracy of
the calculated performance measures. Worksheet 2.6. Data
Integration and Control Findings helps the EQRO review:

WORKSHEET 2.6
Resource for Activity 2, Step 2
Worksheet 2.6 Data Integration and
Control Findings Tool

•

Accuracy of data transfers to the assigned performance
measure repository

•

Accuracy of file consolidations, extracts, and derivations

•

Adequacy of the performance measure data repository to calculate and report performance
measures

•

Management of report production and reporting software

• Guides the EQRO’s review of data
integration and control elements
during the site visit

Step 3: Review Performance Measure Production
The EQRO will review the MCP’s documentation of all steps
undertaken in the production of the performance measures,
including:
•

Data collection from various sources (e.g., membership,
enrollment, provider, claims, or encounter files; medical
records; laboratory, pharmacy, or other ancillary records)

•

Steps taken to integrate the required data into a
performance measure data set or repository

•

Procedures or programs to query the data set/repository to
identify denominators, generate appropriate samples,
determine numerators, and apply proper algorithms to the
data in order to produce valid and reliable performance
measures

WORKSHEET 2.7
WORKSHEET 2.8
Resource for Activity 2, Step 3
Worksheet 2.7. Data and Processes
Used to Produce Performance
Measures: Documentation and Review
Checklist
• Helps the EQRO check the
documentation of steps taken in the
production of the performance
measures
Worksheet 2.8. Data and Processes
Used to Produce Performance
Measures: Findings
• Provides a template for recording
findings based on measurement
plans, policies, and programming
specifications

PROTOCOL TWO | 77

Step 4: Complete the Detailed Review of Measures
In Step 4, the EQRO determines the extent to which the
MCP correctly used the technical specifications to produce
accurate performance measure results. All validation
components should be addressed during this step using
Worksheet 2.2. Performance Measure Validation
Template (or similar tool).
To ensure the integrity and comparability of the
performance measures, the EQRO should pay special
attention to factors affecting the accuracy and
completeness of the denominators and numerators. For
example, the EQRO should assess whether the MCP used
the appropriate data and methods to identify the entire
eligible population for the denominator (including linkage
of data from separate sources, application of inclusions
and exclusions, and creation of complex episodes, where
applicable). In addition, the EQRO should determine
whether the MCP correctly identified and assessed
qualifying medical events for the numerator to include all
appropriate events, while excluding events that do not
qualify. The EQRO should determine whether the
numerators and denominators were calculated
appropriately based on all applicable codes (such as
diagnoses, procedures, and prescription drugs) and all
available data sources (such as membership/enrollment
data, claim/encounter data, provider data, utilization or
medical management information systems data, or data
extracted from medical records).

WORKSHEET 2.2
WORKSHEET 2.9
WORKSHEET 2.10
WORKSHEET 2.11
Resource for Activity 2, Step 4
Worksheet 2.2. Performance Measure
Validation Template
• Provide a template for documenting
audit specifications for the validation
components of each performance
measure listed in Worksheet 2.1.,
and to assess the MCP’s
measurement and reporting process
for each component
Worksheet 2.9. Policies, Data, and
Information Used to Produce Measures:
Review Checklist
• Provides a checklist that tracks
documents and data used to assess
the accuracy of the MCP’s
performance measure calculations
Worksheet 2.10. Measure Validation
Findings
• Provides a template for documenting
adherence to guidance for the
denominator; programming logic,
source code, and calculations;
identifying medical events; exclusion
criteria; population estimates;
identifying the at-risk population;
inclusion of qualifying events in the
numerators; and medical record data
in the numerator
Worksheet 2.11. Interview Guide for
Assessing Processes and Procedures
Used to Produce Numerators and
Denominators
• Provides a list of interview questions
that can be tailored to supplement
findings recorded in Worksheet 2.10
For performance measures requiring
medical record review, please use
Worksheet 2.3. Medical Record Review
Validation Tool. The EQRO should
validate the results of the medical record
review for 30 enrollees who met the
numerator requirements for at least two
measures. For more information, refer to
Activity 1, Step 4.

78 | PROTOCOL TWO

Step 5: Assess the Sampling Process (if applicable)
The EQRO will determine whether the sample represents
the entire eligible population in all relevant dimensions.
The MCP’s sampling method should not exclude any
population subgroups to which the performance measure
applies. For example, when assessing well-child care, the
sample should not exclude children with special health
care needs whose primary care provider is a specialist
other than a pediatrician or family practitioner.

WORKSHEET 2.12
WORKSHEET 2.13
Resource for Activity 2, Step 5
Worksheet 2.12. Policies, Procedures,
and Data Used to Implement Sampling:
Review Checklist

Step 6: Communicate Preliminary Findings
and Outstanding Items

• Guides this review by providing a list
of documents, data, and procedures
to assess the sampling process

At the conclusion of the site visit, the EQRO will
communicate preliminary findings to the MCP, including
any outstanding items for follow-up. The information
communicated during the closing conference will appear
in the EQRO’s subsequent preliminary report to the
MCP. In addition, the EQRO should provide a list of
outstanding items before completing the preliminary
report to allow the MCP the maximum time to resolve
identified issues.

Worksheet 2.13. Sampling Validation
Findings
• For each measure involving a
sample, this worksheet helps assess
the extent to which:
○ The MCP followed the specified
sampling method to produce an
unbiased sample representative of
the entire included population
○ The MCP maintains its
performance measurement
population sample frame to allow
for a sample to be re-drawn or
used as a source for replacement
○ Sample sizes collected conform to
the methodology in the
performance measure
specifications
○ The sample is representative of
the entire population
○ Proper substitution methodology is
followed for performance
measures that include medical
record reviews

PROTOCOL TWO | 79

Activity 3: Conduct Post-Site Visit Activities
Post-site visit activities focus on assessing MCP
corrective actions and reporting findings to the state using
the format and timeframes established by the state. The
EQRO will:
1. Analyze all data and submit a preliminary report to the
MCP detailing areas of concern, suggested methods
for correction, and a timeline for the MCP to make
corrections
2. Re-validate selected performance measures and the
measurement processes the MCP used to make
corrections
3. Re-evaluate the corrected information and submit a
report of validation findings to the state
4. Determine preliminary validation findings for each
measure
5. Assess accuracy of MCP’s performance measure
reports to the state, and
6. Submit the validation report to state
Note that throughout this EQR-related activity or during
any part of an EQR, the state may decide that immediate
corrective action is required.

Step 1: Determine Preliminary Validation
Findings for Each Measure

WORKSHEET 2.3
WORKSHEET 2.6
WORKSHEET 2.10
WORKSHEET 2.13
Resource for Activity 3
Information gathered in Activities 1 and
2 using the following worksheets and
tools may be helpful when preparing the
final validation report:
Worksheet 2.3. Medical Record Review
Validation Template
• Describes procedures and sample
tools for validating medical review
findings
Worksheet 2.6. Data Integration and
Control Findings Tool
• Provides a template for recording
findings from interviews with data
integration and control MCP
personnel
Worksheet 2.10. Measure Validation
Findings
• Provides a template for recording
findings from the measures record
validation review
Worksheet 2.13. Sampling Validation

In the preliminary validation findings report, the EQRO
Findings
will document findings, identify areas of concern, and
• Provides a template to record
make suggestions for corrective action or long-term
findings from the sampling
assessment process
improvement for each of the performance measures the
EQRO validated. The report should indicate which MCP
Worksheet 2.14. Framework for
Summarizing Information About
performance measures and elements of the measures were
Performance Measures
invalid and therefore, should not be reported (if any). The
• Provides a template for summarizing
report should also provide the MCP with correctional
information about performance
guidance for improving the overall measure production
measure results
process. In addition to communicating written findings,
the EQRO may participate in meetings with key MCP personnel responsible for the calculation

80 | PROTOCOL TWO

and reporting of performance measures to assist the MCP with implementing recommended
corrective action.
Once the EQRO has submitted its preliminary findings to the MCP, the MCP may offer
comments and documentation to correct errors and omissions in the EQRO’s preliminary report.
At the discretion of the state, the MCP may recalculate performance measures based on the
findings. The EQRO must then revalidate the revised performance measure(s) and incorporate
the MCP’s comments or revised performance measure validation findings. If the state chooses
not to allow measure re-validation, the recommendations will be reviewed in the following year
as part of the MCP assessment of progress toward recommended improvements.

Step 2: Assess and Document the Accuracy of Performance Measure Reports
The EQRO will assess and document the extent to which the MCP reported the calculated
performance measures correctly in its final report to the state, and verify the reporting of each
performance measure by reviewing:
•

Procedures for submitting reports that meet state requirements (such as specified format,
supporting documentation, and timing)

•

Documentation that the MCP appropriately implemented procedures to properly submit
required reports to state

Step 3: Submit Validation Report to the State
The EQRO will always use the state’s decision rules for
determining the degree to which each of the MCP’s reported
performance measures are accurate and complete. The
decision rules for compliance should be consistent across
MCPs within the state. The final report should follow the
state’s required format, and include the following elements:
•

A list of the measures validated by the EQRO

•

A description of the EQRO’s validation activities
including:

•

The EQRO team members involved in the validation

The data collection methods and analysis

Other considerations relevant to the site visit process

A summary of the validation strategy

List of site visit participants (EQRO, MCP, and vendor)

WORKSHEET 2.14
Resource for Activity 3, Step 3
Worksheet 2.14. Framework for
Summarizing Information About
Performance Measures
• Provides a template for summarizing
information about performance
measure results

Worksheets, tools, and other supporting documentation

PROTOCOL TWO | 81

•

Analyses and conclusions based on the validation process for each performance measure
including:

The validation status of each performance measure (including the results of the medical
record review)

Actual results of the performance measures (not just the results of the validation)

o
•

Recommendations for improving the process for calculating and reporting performance
measures, including implications for the MCP’s data systems, methods, and staffing (e.g.,
programming and analytic capacity)

When possible, the validation report should also identify recommendations from the previous
year’s report submitted to the state, and discuss progress made on these recommendations over
the past year based on information gathered during the validation process.
END OF PROTOCOL 2

82 | PROTOCOL TWO

Worksheets for Protocol 2: Performance Measure Validation
Tools
Instructions. Use these or similar worksheets to assist in validating performance measures reported by the MCP.
These worksheets provide templates for validating performance measures and a framework for summarizing
information about performance measures in the EQR technical report. This tool includes the following worksheets
crosswalked to the applicable Activity and Step:
Worksheet Name

Protocol Activity and Step

Worksheet 2.1. List of Performance Measures to be
Validated

Activity 1. Step 1. Define the Scope of the Validation

Worksheet 2.2. Performance Measure Validation
Template

Activity 1. Step. 1. Define the Scope of the Validation
Activity 1. Step 3. Conduct Detailed Review of Measures
Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.3. Medical Review Validation Template

Activity 1. Step 4. Initiate Review of Medical Record Data
Collection
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.4. Potential Documents and Process for
Review

Activity 1. Step 5. Prepare for the MCP Site Visit

Worksheet 2.5. Interview Guide for MCP Data Integration Activity 2. Step 1. Review Information Systems
and Control Personnel
Underlying Performance Measurement
Worksheet 2.6. Data Integration and Control Findings
Tool

Activity 2. Step 2. Assess Data Integration and Control
for Performance Measure Calculation
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.7. Data Processes Used to Produce
Performance Measures: Documentation and Review
Checklist

Activity 2. Step 3. Review Performance Measure
Production

Worksheet 2.8. Data and Processes Used to Produce
Performance Measures: Findings

Activity 2. Step 3. Review Performance Measure
Production

Worksheet 2.9. Polices, Data, and Information Used to
Produce Measures: Checklist

Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.10. Measure Validation Findings

Activity 2. Step 4. Complete the Detailed Review of
Measures
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.11. Interview Guide for Assessing
Processes Used to Produce Numerators and
Denominators

Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.12. Policies, Procedures, and Data Used to Activity 2. Step 5. Assess the Sampling Process (if
Implement Sampling: Review Checklist
applicable)
Worksheet 2.13. Sampling Validation Findings

Activity 2. Step 5. Assess the Sampling Process (if
applicable)
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.14. Framework for Summarizing
Information about Performance Measures

Activity 3. Step 3. Conduct Post-Site Visit Activities

PROTOCOL TWO | 83

For each MCP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
EQRO interview date
Type of delivery system (check all that apply)

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM entity □ LTSS
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

84 | PROTOCOL TWO

Worksheet 2.1. List of Performance Measures to be Validated
Instructions. This worksheet is used to identify the measures to be validated, the data source, reporting frequency,
and format as described in Activity 1. Step 1. Complete the worksheet for each measure to be validated, and adapt
as needed. The list below is illustrative of the performance measures that could be included in the worksheet.

Performance
Measures
(Illustrative)
Childhood
Immunization
Status (CISCH) (NCQA):
Combo 3

NQF #

Admin.
Data

Medical
Record
Review
(MRR)

Hybrid
(Admin.
Data and
MRR)

Electronic
Health
Record

Survey

Reporting
Frequenc
y and
Comment
Format
s

0038

Immunizations 1407
for Adolescents
(IMA-CH)
(NCQA):
Combo 1
Well-Child
1392
Visits in the
First 30 Months
of Life (W30CH) (NCQA)
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 85

Worksheet 2.2. Performance Measure Validation Template
Instructions. For each performance measure, use this template to gather audit specifications for the validation
components (as described in Activity 1. Steps 1 and 3, and Activity 2. Step 4) and to assess the MCP’s measurement
and reporting process for each component.
For each validation component, indicate whether the measure meets validation requirements by checking “Yes,” “No,”
or “Not applicable.” Insert comments to explain “Not met” and “Not applicable” responses. Use the following guidance
to assess each component.
•

Yes: The MCP’s measurement and reporting process was fully compliant with state specifications

•

No: The MCP’s measurement and reporting process was not fully compliant with state specifications. This
designation should be used for any validation component that deviates from the state specifications, regardless of
the impact of the deviation on the final rate. All components with this designation must include an explanation of
the deviation in the comments section

•

Not applicable: The validation component was not applicable. Include an explanation in the comments section
(e.g., sampling not required, medical record review not included)

Managed Care Plan __________________________________________________________________
Performance Measure __________________________________________________________
Method for Calculating Measure: [ ] Admin

Validation component
Documentation:
Did appropriate and complete measurement
plans and programming specifications exist,
including data sources, programming logic,
and computer source code?
Were internally developed codes used?
Denominator:
Were all the data sources used to calculate
the denominator complete and accurate
(e.g., eligibility files, claims files/encounter
data, medical records, provider files,
pharmacy records, including those for
members who received services outside the
MCP’s network)?

86 | PROTOCOL TWO

[ ] Medical Record Review [ ] Hybrid [ ] EHR [ ] Survey
Audit
specifications

Yes

Not
applicable

Comments

Validation component

Audit
specifications

Yes

Not
applicable

Comments

Did the calculation of the performance
measure adhere to the specifications for all
components of the denominator (e.g.,
member ID, age, sex, continuous
enrollment, clinical codes such as ICD-10,
CPT® 51, DRGs, member months/member
years, and adherence to the measurement
period)?
Numerator:
Were the data sources used to calculate the
numerator complete and accurate (e.g.,
claims files, medical records, provider files,
pharmacy records, including those for
members who received services outside the
MCP’s network)?
Did the calculation of the performance
measure adhere to the specifications for all
components of the numerator (e.g., member
ID, clinical codes such as ICD-10, CPT®,
LOINC, DRGs, pharmacy data, relevant
time parameters such as
admission/discharge dates or treatment
start and stop dates, adherence to the
measurement period, number or type of
provider)?
If medical record abstraction was used,
were the abstraction tools adequate?
If the hybrid method was used, was the
integration of administrative and medical
record data adequate?
If the hybrid method or medical record
review was used, did the results of the
medical record review validation
substantiate the reported numerator?
Sampling:
Was the sample unbiased?
Did the sample treat all measures
independently?
Did the sample size and replacement
methodologies meet specifications?

PROTOCOL TWO | 87

Validation component

Audit
specifications

Yes

Not
applicable

Comments

Reporting:
Were the state specifications for reporting
performance measures followed?
Overall assessment: In the comments
section, note any recommendations for the
MCP’s measurement and reporting process

Yes
(Please
explain)

Additional Audit Questions
Were any members excluded for contraindications found in the administrative data?
Were any members excluded for contraindications found during the medical record
review?
Were internally developed codes used?

What is the estimated impact of data incompleteness
on the rate(s) calculated for this measure?
• 0–5 percentage points
• >5–10 percentage points
• >10–20 percentage points
• >20–40 percentage points
• >40 percentage points
• Unable to determine
What is the direction of the bias? (Check one)
• Over-reporting
• Under-reporting
• Not applicable (no bias detected)
What documentation was used to estimate the above
percentage (e.g., internal reports, studies, comparison to
medical records)?

88 | PROTOCOL TWO

Check
one

Comments

Overall Validation Finding
Provide an overall validation finding for each performance measure. The validation finding is determined by the
magnitude of the errors detected for the audit elements, not by the number of audit elements determined as “NO.”
Consequently, it is possible that an error for a single audit element may result in a designation of “Do Not Report”
(DNR) because the impact of the error materially biased the reported performance measure. Conversely, it is also
possible that several audit element errors may have little impact on the reported rate and, thus the measure is
“Reportable” (R).
Performance Measure Validation Finding (check one)

Comments

[ ] R = Reportable; measure was compliant with state
specifications
[ ] DNR = Do not report; MCP rate was materially biased and
should not be reported
[ ] NA = Not applicable; the MCP was not required to report the
measure
[ ] NR = Measure was not reported because the MCP did not
offer the required benefit
Overall assessment: In the comments section, note justification
and recommendations for the validation finding.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 89

Example of Worksheet 2.2. Performance Measure Validation Template
Below is an example of a completed, customized performance measure validation worksheet similar to what an
EQRO would prepare before its site visit. This worksheet is based on the Child and Adult Core Set specifications for
the performance measure.
Performance Measure: Chlamydia Screening in Women Ages 16 to 20 (CHL-CH) and Ages 21 to 24 (CHL-AD)
(NCQA)_____
Method for Calculating Measure: [ X ] Admin

Validation component

[ ] Medical Record Review [ ] Hybrid [ ] EHR [ ] Survey

Audit specifications

Yes

Documentation:
Did appropriate and complete
• Obtain and review all file X
layouts, code,
measurement plans and
documentation
programming specifications exist,
including data sources,
• Code and documentation
programming logic, and computer
mapped to measure
specification
source code?
Denominator:
Were all the data sources used to
calculate the denominator
complete and accurate (e.g.,
eligibility files, claims files,
provider files, pharmacy
records)?

90 | PROTOCOL TWO

Not
applicable

Comments

Validation component

Audit specifications

• Medicaid population
appropriately segregated
from commercial and
Medicare

Yes

Not
applicable

Comments

• Population defined as
active Medicaid
enrollment as of 12/31 of
measurement year
• Members ages 21-24 as
of 12/31 of the
measurement year
• Only females selected
• Members enrolled in
MCP on 12/31 of the
measurement year
• Continuously enrolled
from 1/1 to 12/31 of the
measurement year with
no more than one break
of up to 45 days allowed
• Shifts between Medicaid
and CHIP enrollment
were not counted as
breaks; shifts between
Medicaid and CHIP and
commercial enrollment
were counted as breaks.

Numerator:
Were the data sources used to
• Sexually active based on
pharmacy and
calculate the numerator complete
claims/encounter data
and accurate (e.g., claims files,
medical records, provider files,
• Properly identified
pharmacy records, including
enrollees. Based on the
ISCA findings, the data
those for members who received
sources used for the
services outside the MCP’s
numerator were accurate
network)?

PROTOCOL TWO | 91

Validation component

Audit specifications

Did the calculation of the
performance measure adhere to
the specifications for all
components of the numerator
(e.g., member ID, clinical codes
such as ICD-10, CPT®, LOINC,
DRGs, pharmacy data, relevant
time parameters such as
admission/discharge dates or
treatment start and stop dates,
adherence to the measurement
period, number or type of
provider)?

• Exclusions were
performed according to
state specifications

Yes

Not
applicable

Comments

• Only the codes listed in
specifications as defined
by state were counted as
exclusions
• Standard codes listed in
state specifications
(and/or properly mapped
internally developed
codes) were used
• Members were counted
only once; double
counting was prevented
• Service performed
between 1/1 and 12/31 of
the measurement year

If medical record abstraction was
used, were the abstraction tools
adequate?

Not applicable; no
medical record
abstraction

If the hybrid method was used,
was the integration of
administrative and medical record
data adequate?

Not applicable; hybrid
method not specified
for this measure

If the hybrid method or medical
record review was used, did the
results of the medical record
review validation substantiate the
reported numerator?

Not applicable; hybrid
method not specified
for this measure

Did the sample treat all measures
independently?

Not applicable; no
sampling

Did the sample size and
replacement methodologies meet
specifications?

Not applicable; no
sampling

Sampling:

92 | PROTOCOL TWO

Validation component

Audit specifications

Yes

Not
applicable

Comments

Reporting:
Were the state specifications for
reporting performance measures
followed?

• Measure-eligible
population is accurate
and documented
(inclusions, exclusions)

• Method is accurate and
documented
(measurement period,
data source)
• Information on numerator,
denominator, rate is
accurate and
documented
• Deviations (if any) are
accurate and
documented
Overall assessment: In the
comments section, note any
recommendations for the MCP’s
measurement and reporting
process

Additional Audit Questions

Measure meets all
audit specifications
and is reportable by
the state

Yes
(Please
explain)

Were any members excluded for contraindications found in the administrative data?

Were any members excluded for contraindications found during the medical record
review?

X (Not
applicable)

Were internally developed codes used?

What is the estimated impact of data incompleteness on the rate(s) calculated
for this measure? (Check one)
• 0–5 percentage points

• >5–10 percentage points
• >10–20 percentage points
• >20–40 percentage points
• >40 percentage points
• Unable to determine
What is the direction of the bias? (Check one)
• Over-reporting
• Under-reporting

PROTOCOL TWO | 93

• Not applicable (no bias detected)

What documentation was used to estimate the above percentage (e.g., internal
reports, studies, comparison to medical records)?

Internal reports

Comments

[ X ] R = Reportable; measure was compliant with state
specifications
[ ] DNR = Do not report; MCP rate was materially biased and
should not be reported
[ ] NA = Not applicable; the MCP was not required to report the
measure
[ ] NR = Measure was not reported because the MCP did not
offer the required benefit
Overall assessment: In the comments section, note justification
and recommendations for the validation finding.
Performance Measure Validation Finding

Adhered to all specifications and no data
concerns detected.

94 | PROTOCOL TWO

Worksheet 2.3. Medical Record Review Validation Template
Instructions. This template provides instructions for conducting the medical record review (as described in Activity 1.
Step 4 and Activity 3) and two tables to summarize re-abstraction findings from the review (Table 1) and record the
impact of findings from the review (Table 2).
The purpose of medical record review (MRR) validation is to verify the accuracy of the MRR conducted by each MCP.
For each of at least two measures that included medical record review, the EQRO will validate the medical records of
30 enrollees found to meet numerator requirements. In states with separate Medicaid and CHIP programs, the EQRO
will review 30 enrollees in each CHIP MCP and 30 enrollees in each Medicaid MCP for each of at least two measures
that included medical record review. Only those members included in a hybrid sample will be selected—the EQRO
will not conduct medical record audits to validate administrative data.
For each measure in which medical record review was used, the EQRO will request a list of all of the members in the
MCP’s MRR sample. From that list:
•

The EQRO will identify a sample of 30 members who meet numerator requirements

•

MCPs will then be asked to provide access to or copies of medical records so that the EQRO can verify that each
member was appropriately included in the denominator and received the required numerator service(s)

•

In cases where there are fewer than 30 numerator positives, the EQRO will review all records for that measure

To provide sufficient time for each MCP to gather the required medical record documentation, the EQRO may direct
the MCPs to submit their lists of members in their hybrid sample twice— the first list as a preliminary submission and
the second list as a final submission:
•

Submitting a first list before completion of the MRR process would allow an MCP additional time to retrieve
medical record documentation

•

Soon after receipt of the first list, the EQRO will provide the MCP with the list of medical records for which
documentation must be submitted

•

Only a portion of the 30 medical records for the validation sample will be included in the EQRO’s first sample
request list

•

The remainder of the 30 records will be selected from the final list. While the first submission of MRR findings is
optional, it is recommended

The EQRO should accept the first list submission approximately one month before the scheduled audit or another
time specified by the EQRO. If an MCP chooses to submit a first list of medical records, it must still submit a final
listing sufficiently in advance of the scheduled audit as directed by the EQRO. For each submission:
•

MCPs will need to identify all members for whom MRR has been conducted and indicate which members have
been found to be numerator positives through MRR

•

The final list must reflect the MCP’s final medical record review findings, with members for whom a medical record
was never found identified as not having met the numerator requirements

No predetermined “passing” grade is set for the medical record audit. Rather, auditors will use the MRR results to
determine if the hybrid rate (or solely MRR rate if applicable) is biased, and to what extent that bias affects the final
reported rate for that measure. The EQRO will identify to the state what effects bias, as well as incomplete data, will
have on the MCP’s calculation of the performance measure. For each of the evaluated measures, auditors will
determine the impact of the findings from the MRR validation process on the MCP’s Final Audit Designation.

PROTOCOL TWO | 95

Step 1: Calculate the Medical Record Review Error Rate
The EQRO will review up to 30 records identified by the MCP as meeting numerator requirements (as determined
through MRR) for the measures audited. Records are randomly selected from the entire population of MRR
numerator positives identified by the plan, as indicated on the MRR numerator listings submitted to the EQRO:
•

If fewer than 30 medical records are found to meet numerator requirements, all records are reviewed

•

Administrative numerator positives are not included as part of this validation process

The EQRO will calculate a MRR error rate for each performance measure calculated by the hybrid method or solely
from MRR as illustrated in Table 1.
Table 1: Summary of Medical Record Review (MRR) Re-abstraction Findings
Column A

Column B

Performance
measure

Number of
MMR positives
selected for
audit

Column C
Number of
medical
records
received

Column D
Number of
medical
records found
compliant

Column E

Column F

Accuracy rate
(%)

Error rate (%)

(col. D/col. B)

(100% - col. E)

Column A: Name of performance measure
Column B: Total number of MRR numerator positive records re-abstracted by EQRO as part of the medical record
review validation process (i.e., 30, or the total population if less than 30 MRR numerator positives were
reported)
Column C: Total number of medical records submitted to EQRO as part of the medical record review validation
process (i.e., should be equal to Column B or less than Column B if one or more records were not
submitted on time)
Column D: Total number of medical records reviewed by EQRO and identified as meeting numerator requirements
Column E: Accuracy rate = percent of records selected for audit that were identified as meeting numerator
requirements (Column D/Column B)
Column F: Error rate = percent of records selected for audit that were identified as not meeting numerator
requirements (100% - Column E)

Step 2: Determining the Potential Impact of Medical Record Review Re-abstraction Findings on
Final Audit Designations
The next step in MRR validation is to determine whether any medical record review errors significantly biased the
final reported rate for a given performance measure. To make this determination, the EQRO, as directed by the state,
should develop and follow decision rules such as the following:
Sample Decision Rules:
•

Error Rate of 10 Percent or Less. If the error rate (Table 1, column F) is 10 percent or less, then the measure
automatically passes the MRR validation. The Final Audit Designation is then determined based on the auditors’
findings from the ISCA conducted as Pre-Site Visit Activity 3 and Site Visit Activity 1. As long as no errors leading

96 | PROTOCOL TWO

to significant bias are discovered during the other components of the audit process, the final rate is considered as
having met the validation standards
•

Error Rate of Greater than 10 Percent. If the error rate (Table 1, column F) is greater than 10 percent, then the
auditors determine the impact of the MRR validation findings on the final reported rate for the measure. For each
of the measures under review, auditors evaluate the impact of the MCP’s MRR processes on its final reported
rate by extrapolating findings from the audited medical record sample to the universe of all MRR positives. Details
on this process are in Table 2

•

The maximum amount of bias allowed for the final rate to be considered reportable is “X” percentage
points (to be determined by each state). If the amount of error in the MCP’s MRR process (Table 2, line 8)
does not cause the final reported rate to be biased by more than X percentage points, then the measure passes
the MRR validation. The compliance designation is then determined based solely on the auditors’ findings from
the ISCA. As long as no errors leading to significant bias are discovered during the other components of the
performance measure audit process, the final rate is considered valid

•

If the amount of error in the MCP’s medical review process (Table 2, line 8) ultimately causes the final
reported rate to be biased by more than X percentage points, the rate is automatically considered invalid. The
performance measure is then designated as invalid

Table 2: Impact of MRR Findings
Line #

Description

Final data collection method used (e.g., MRR,
hybrid)

Error rate (percentage of records selected for
audit that were identified as not meeting
numerator requirements, as shown in Table 1,
column F)

Is error rate <10%? (Yes or No)

Measure A

Measure B

Measure C

• If yes, MCP passes MRR validation; no
further MRR calculations necessary
• If no, the full table must be completed to
determine the impact on the final rate
4

Denominator (the total number of members
identified for the denominator of this measure,
as identified by the MCP)

Weight of each medical record (impact of each
medical record on the final overall rate;
determined by dividing 100% by the
denominator in line 4)

Total number of MRR numerator positives
identified by the MCP using MRR

Expected number of false positives
(Estimated number of medical records
inappropriately counted as numerator positives;
determined by multiplying the error rate in line 2
by line 6, the total number of MRR numerator
positives reported)

PROTOCOL TWO | 97

Line #

Description

Estimated bias in final rate

Measure A

Measure B

Measure C

(The amount of bias caused by medical record
review, measured in percentage points;
determined by multiplying the expected number
of false positives in line 7 by line 5, the weight of
each medical record)
•

If line 8 is <=X%, then the final rate is not considered to be significantly biased by MRR alone. If the other
components of the audit process did not identify any other issues that would introduce bias into the rate, the rate
will be considered valid

•

If line 8 is >X%, then the final rate is considered to be significantly biased. The measure will be considered invalid

98 | PROTOCOL TWO

Worksheet 2.4. Potential Documents and Processes for Review
Instructions. To assist the EQRO in assessing the MCP’s information system and validity of reported performance
measures, this worksheet provides a checklist of documents, data, and procedures the MCP should make available
before or during the site visit (as described in Activity 1, Step 5). Record any questions or concerns raised by the
review of documents and/or processes, and any specific checks or tests the EQRO would like to conduct or have
demonstrated during the site visit. The EQRO can use its discretion in selecting which ones to review.
For example:
•

Compare samples of data in the repository to transaction files. Are any members, providers, or services lost in the
process?

•

Is the required level of coding detail maintained (e.g., all significant digits, primary and secondary diagnoses
remain)?

•

If the MCP uses a performance measure repository, review the repository structure. Does it contain all the key
information necessary for performance measure reporting?

•

How does the MCP test the process used to create the performance measure reports?

•

Does the MCP use any algorithms to check the reasonableness of data integrated to report the MCP-level
performance measures?

•

Examine report production logs and run controls. Is there adequate documentation of the performance measure
report generation process? How are report generation programs documented? Is there version control in place?

Checklist of documents and processes for review

Reviewed?
Y/N

Comments for site visit

1. Data integration and control
Procedures and standards for all aspects of the data
repositories used in producing performance measures, including
building, maintaining, managing, and testing performance
measures
Manuals that include application system development
methodology, database development, and design and decision
support system utilization
System documentation including flow charts and codes for
backups, recovery, archiving, and other control functions
Procedures to consolidate information from disparate
transaction files
Record and file formats and descriptions, for entry,
intermediate, and repository files
Electronic formats and protocols
Electronic transmission procedures documentation
Processes to extract information from the repositories
Source code data entry, data transfer, and data manipulation
programs and processes
Descriptive documentation for data entry, transfer, and
manipulation programs and processes

PROTOCOL TWO | 99

Checklist of documents and processes for review
If applicable, procedures for coordinating vendor activities to
safeguard the integrity of the performance measurement data
Samples of data from repository and transaction files to assess
accuracy and completeness of the transfer process
Comparison of actual results from file consolidation and data
abstracts to those which should have resulted according to
documented algorithms
Documentation of data flow among vendors to assess the
extent to which there was proper implementation of procedures
to safeguard the integrity of the performance measure data
Documentation of data cut-off dates
Documentation of proper run controls and of staff review of
report runs
Copies of files and databases used for performance measure
calculation and reporting
Procedures governing production process for MCP performance
measures, including standards and schedules
2. Collection, calculation, and documentation of
performance measures
Policies for the documentation of data requirements, data
issues, validation efforts, and results
A project or measurement plan for each performance measure
Documentation of programming specifications, including work
flow, data sources, and uses which include diagram or narrative
descriptions
Documentation of the original universe of data that includes
record-level patient identifiers, which can be used to validate
programming logic for creating denominators, numerators, and
samples
Documentation of computer queries, programming logic, or
source code used to create final denominators, numerators, and
interim data files
Documentation that includes dated job log or computer run for
denominators and numerators, with record counts for each
programming step and iteration
Documentation of medical record review including:
• Qualifications of medical record review supervisor and staff
• Reviewer training materials
• The use of audit tools, including completed copies of each
record-level reviewer determination
• All case-level critical performance measure data elements
used to determine a positive or negative event or exclude a
case, and
• Interrater reliability testing procedures and results

100 | PROTOCOL TWO

Reviewed?
Y/N

Comments for site visit

Checklist of documents and processes for review

Reviewed?
Y/N

Comments for site visit

Documentation of statistical test results and any corrections or
adjustments to data along with justification for such changes
Documentation of sources of any supporting external data or
prior years’ data used in reporting
Policies to assign unique membership ID that allows all services
to be properly related to the specific appropriate enrollee,
despite changes in status, periods of enrollment or
disenrollment, or changes across product lines (e.g., CHIP and
Medicaid).
Procedures to identify, track, and link member enrollment by
product line, product, geographic area, age, sex, member
months, and member years
Procedures to track individual members through enrollment,
disenrollment, and possible re-enrollment
Procedures to track members through changes in family status,
changes in benefits or managed care type (if they switch
between Medicaid coverage and another product within the
same MCP)
Methods to define start and cessation of coverage
Procedures to link member months to member age
Description of software or programming languages used to
query each database
Description of software used to execute sampling of population
files when sampling is used
Member database
Provider data (including facilities, labs, pharmacies, physicians,
etc.)
Database record layout and data dictionary
Survey data used for performance measures (See Protocol 6)
Policies to maintain files from which the samples are drawn in
order to keep population intact in the event that a sample must
be re-drawn, or replacements made
Computer source code or logic identifying specified sampling
techniques and documentation that the logic matches the
specifications set forth for each performance measure, including
sample size and exclusion methodology
Methods used for sampling for measures calling for medical
record or hybrid data
Documentation assuring that sampling methodology treats all
measures independently and that there is no correlation
between drawn samples
Observation or documentation of procedures in which a biased
sample was identified and corrected

PROTOCOL TWO | 101

Checklist of documents and processes for review
Documentation of “frozen” or archived files from which the
samples were drawn, and if applicable, documentation of the
MCP’s process to re-draw a sample or obtain necessary
replacements
For performance measures that are easily under-reported,
procedures to capture data that may reside outside the MCP’s
datasets
Procedures for mapping non-standard codes to standard coding
Policies, procedures, and materials that provide evidence of
proper training, supervision, and adequate tools for medical
record abstraction tasks (this may include medical record
abstraction tools, training material, checks of inter-rater
reliability, etc.)
Procedures for assuring that combinations of record-review data
with administratively determined data are consistent and
verifiable
Evidence that MCP’s use of codes to identify medical events
were correctly evaluated when classifying members for
inclusion or exclusion in the numerator
Evidence that MCP has counted each member and/or event
only once
Programming logic or demonstration that confirms that any nonstandard codes used in determining the numerator have been
mapped to a standard coding scheme in a manner that is
consistent, complete, and reproducible
Programming logic or source code that identifies the process for
integrating administrative and medical record data for numerator
Procedures for properly executing complex medical algorithms,
such as
• Claim-dependent events
• Events that require matching claims and pharmacy data
• Events that require matching visit codes, and
• Events that require accurately identifying and computing
multiple numerator events
Procedures for displaying denominator counts, numerator
counts, precision levels, sums and cross-totals
Procedures for reporting small sample sizes (to be consistent
with required methodology established by state)
Programming logic and/or source code for arithmetic calculation
of each measure
Review of reported measures to assess consistency of common
elements (e.g., membership counts, number of pregnancies and
births, etc.)

102 | PROTOCOL TWO

Reviewed?
Y/N

Comments for site visit

Checklist of documents and processes for review

Reviewed?
Y/N

Comments for site visit

Programming logic and/or source code for measures with
complex algorithms, to ensure adequate matching and linkage
among different types of data
Documentation showing confidence intervals of calculations
when sampling methodology used
Documentation showing calculation of levels of significance of
changes
Procedures for submitting reports that meet state requirements
(e.g., specified electronic format, supporting documentation,
and timing)
Documentation that procedures for properly submitting required
reports to state were implemented appropriately
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 103

Worksheet 2.5. Interview Guide for MCP Data Integration and Control Personnel
Instructions. As part of the EQRO’s review of information system components the MCP uses to produce
performance measures, the EQRO should interview key staff (including appropriate vendor staff) involved in the
production of performance measures using questions tailored to the MCP’s processes for producing those measures.
These interviews are an opportunity to supplement the review of information system policies, procedures, and data
(as described in Activity 2, Step 1). Please tailor the questions below as appropriate.
MCP Contact and Background Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, email address, and date of interview, if applicable.
MCP name:
MCP contact name(s):
Title(s):
Mailing address:
Phone number(s):
Email address:
Interview Date:
EQRO reviewers:
Year of first Medicaid enrollment:
Year of first CHIP enrollment:
Year of first MCP performance report
(any product line)
1. Has the MCP previously undergone validation of its state performance measure reporting process? If so, when
did the validation take place and who conducted it?
2. How is performance measure data collection accomplished? (Check all that apply)
[

] By querying the applicable information system on-line

[ ] By using extract files created for analytical purposes? If so, how frequently are the files updated? How
do they account for claim/ encounter for accuracy?
[ ] By using a separate relational database or data warehouse? If so, is this the same system from which
all other reporting is produced?
[ ] Reports created from an NCQA-certified vendor software product? If so, how frequently are the files
updated? How are reports checked for accuracy?
3. Review the procedure(s) for consolidating claims/encounter, member, provider, and other data necessary for
performance reporting (whether it be into a relational database or file extracts on a measure-by-measure basis):
• How many different sources of data are merged together to create reports?
• What control processes are in place to ensure that this merger is accurate and complete?

104 | PROTOCOL TWO

4. How does the MCP test the process used to create the performance measure reports?
5. Does the MCP use any algorithms to check the reasonableness of data integrated to report the MCP performance
measures?
6. Is performance measurement reporting programming reviewed by supervisory staff?
7. Please describe any internal backup for performance measure programmers, if one exists. Do others know the
programming language and the structure of the actual programs? Please describe what documentation exists, if
any.
8. How does the MCP prevent loss of claim and encounter data when systems fail?
9. Please describe the administrative data backup systems are in place.
10. What types of authorization are required to be able to access claims/encounter, provider, membership, and
performance measure repository data?
11. Please describe documentation review and demonstrations provided.

PROTOCOL TWO | 105

Worksheet 2.6. Data Integration and Control Findings Tool
Instructions. During the site visit (described in Activity 2, Step 2 and Activity 3), this tool helps the EQRO review the:
•

Accuracy of data transfers to the assigned performance measure repository

•

Accuracy of file consolidations, extracts, and derivations

•

Adequacy of the performance measure data repository to calculate and report performance measures, and

•

Management of report production and reporting software

For each data integration and control element, please indicate whether it was met, not met, or not applicable (N/A),
and any relevant comments.
1. Accuracy of data transfers to assigned
performance measure repository

Met

Not
met

N/A

Comments

Met

Not
met

N/A

Comments

MCP processes accurately and completely transfer
data from the transaction files (e.g., membership,
provider, encounter/claims) into the repository used
to keep the data until the calculations of the
performance measures have been completed and
validated
Samples of data from repository are complete and
accurate

2. Accuracy of file consolidations, extracts, and
derivations
MCP’s processes to consolidate diversified files and
to extract required information from the performance
measure repository are appropriate
Actual results of file consolidations or extracts were
consistent with those which should have resulted
according to documented algorithms or
specifications
Procedures for coordinating the activities of vendors
ensure the accurate, timely, and complete
integration of data into the performance measure
database
Computer program reports or documentation reflect
vendor coordination activities, and no data elements
needed for performance measure reporting are lost
or inappropriately modified during transfer

106 | PROTOCOL TWO

3. If the MCP uses one, the structure and format
of the performance measure data repository
facilitates any required programming necessary
to calculate and report required performance
measures

Met

Not
met

Met

Not
met

N/A

Comments

The repository’s design, program flow charts, and
source codes enable analyses and reports
Proper linkage mechanisms have been employed to
join data from all necessary sources (e.g.,
identifying a member with a given
disease/condition)

4. Assurance of effective management of report
production and of the reporting software

N/A

Comments

Documentation governing the production process,
including MCP production activity logs, and MCP
staff review of report runs was adequate
Prescribed data cutoff dates were followed
The MCP has retained copies of files or databases
used for performance measure reporting, in the
event that results need to be reproduced
Reporting software program is properly documented
with respect to every aspect of the performance
measurement reporting repository, including
building, maintaining, managing, testing, and report
production
MCP’s processes and documentation comply with
the MCP standards associated with reporting
program specifications, code review, and testing
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 107

Worksheet 2.7. Data and Processes Used to Produce Performance Measures:
Documentation Review Checklist
Instructions. During the site visit, this tool helps the EQRO check the documentation of steps taken in the production
of the performance measures. It is intended to guide document review (as described in Activity 2, Step 3).

Documentation
Documentation of overall policies and
procedures
Policies which stipulate and enforce
documentation of data requirements, issues,
validation efforts and results
Procedures for displaying denominator counts,
numerator counts, precision levels, and totals
Procedures for reporting small sample sizes
(consistent with state’s required methodology)
All reported measures to assess consistency of
common elements (e.g., membership counts,
number of pregnancies and births, etc.)
Documentation for each measure:
Programming logic and/or source code for
arithmetic calculation
A project or measurement plan, including work
flow
Documentation of programming specifications
and data sources
Documentation of the original universe of data
including record-level patient identifiers that can
be used to validate entire programming logic for
creating denominators, numerators, and
samples
Documentation of computer queries,
programming logic, or source code used to
create denominators, numerators, and interim
data files
Documentation of medical record review for
each measure, as appropriate, including:
qualifications of medical record review
supervisor and staff; reviewer training materials,
audit tools used (including completed copies of
each record-level reviewer determination), all
case-level critical performance measure data
elements used to determine a positive or
negative event or exclude a case from same,
and inter-rater reliability testing procedures and
results

108 | PROTOCOL TWO

Not
Not
Reviewed Reviewed applicable

Comments

Documentation

Not
Not
Reviewed Reviewed applicable

Comments

Documentation of results of statistical tests and
any corrections or adjustments to data along
with justification for such changes for each
measure, as appropriate
Documentation showing calculation of levels of
significance of changes for each measure
Documentation showing confidence intervals of
calculations when sampling methodology used
Documentation of sources of any supporting
external data or prior years’ data used in
reporting
Overall assessment: In the comments section,
note:
• How are policies governing documentation of
data requirements for performance
measurement (e.g., data file and field
definitions, mapping between standard and
non-standard codes) updated and enforced?
Who is responsible for this?
• How are programming specifications for
MCP performance measures documented?
Who is responsible for this?
• Are documentation processes up to date?
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 109

Worksheet 2.8. Data and Processes Used to Produce Performance Measures: Findings
Instructions. Record findings based on measurement plans, policies, and programming specifications, as described
in Activity 2, Step 3.
1. Measurement plans and policies that stipulate and
enforce documentation of data requirements, issues,
validation efforts, and results. These include the
following audit elements

Met

Not
Met

N/A

Comments

Met

Not
Met

N/A

Comments

Data file and field definitions used for each measure
Maps to standard coding if not used in original data
collection
Statistical testing of results and any corrections or
adjustments made after processing
Overall assessment: In the comments section, note any
recommendations for improving measurement plans and
policies

2. Documentation of programming specifications
(which may be either a schematic diagram or in
narrative form) for each measure includes at least the
following audit elements:
All data sources, including external, supplemental data
(whether from a vendor, public registry, or other outside
source), and any prior year data, if applicable
A project or measurement plan, including workflow
Detailed medical record review methods and practices,
including:
• The qualifications of medical record review supervisor
and staff
• Reviewer training materials
• Audit tools used (including completed copies of each
record-level reviewer determination)
• All case-level critical performance measure data
elements used to determine a positive or negative
event or exclude a case from same, and
• Inter-rater reliability testing procedures and results
Detailed computer queries, programming logic, or source
code used to identify the population or sample for the
denominator and/or numerator
Documentation of the original universe of data including
record-level patient identifiers that can be used to validate
entire programming logic for creating denominators,
numerators, and samples

110 | PROTOCOL TWO

2. Documentation of programming specifications
(which may be either a schematic diagram or in
narrative form) for each measure includes at least the
following audit elements:

Met

Not
Met

N/A

Comments

If sampling is used, a description of sampling techniques
and documentation assuring the reviewer that samples
used for baseline and repeat measurements of the
performance measures were chosen using the same
sampling frame and methodology
Documentation of calculation for changes in performance
from previous periods, if applicable, including statistical
tests of significance
Data that are related from measure to measure are
consistent (e.g., membership counts, provider totals,
number pregnancies and births)
Appropriate statistical functions are used to determine
confidence intervals when sampling is used in the
measure
When determining improvement in performance between
measurement periods, appropriate statistical methodology
is applied to determine levels of significance of changes
Overall assessment: In the comments section, note any
recommendations for improving programming
specifications for each performance measure
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 111

Worksheet 2.9. Policies, Procedures, and Data Used to Produce Performance Measures:
Review Checklist
Instructions. Use this checklist to track documents and data used to assess the accuracy of the MCP’s performance
measure calculations (as described in Activity 2, Step 4).

Policies, Procedures, and Data to be Reviewed
Policies to assign unique membership ID that allows
all services to be properly related to the specific
appropriate enrollee, despite changes in status,
periods of enrollment or disenrollment, or changes
across product lines (e.g., Medicare and Medicaid)
Procedures to identify, track, and link member
enrollment by product line, product, geographic area,
age, gender, member months, member years
Procedures to track individual members through
enrollment, disenrollment, and possible re-enrollment
Procedures to track members through changes in
family status, changes in employment or benefits or
managed care type (if they switch between Medicaid
coverage and another product within the same MCP)
Methods to define start and cessation of coverage
Procedures to link member months to member age
Description of software or programming languages
used to query each database
Programming logic and/or source code for arithmetic
calculation of each measure
Programming logic and/or source code for measures
with complex algorithms, to ensure adequate matching
and linkage among different types of data
Member database
Provider data (including facilities, labs, pharmacies,
physicians, etc.)
Database record layout and data dictionary
Survey data
For performance measures which are easily underreported, procedures to capture data that may reside
outside the MCP’s data sets
Procedures for mapping non-standard codes to
standard coding to ensure consistency, completeness,
and reproducibility
Policies, procedures, and materials that evidence
proper training, supervision, and adequate tools for
medical record abstraction tasks (may include medical

112 | PROTOCOL TWO

Reviewed

Not
Reviewed

Comments

Policies, Procedures, and Data to be Reviewed

Reviewed

Not
Reviewed

Comments

record abstraction tools, training material, checks of
inter-rater reliability, etc.)
Procedures for assuring that combinations of recordreview data with administratively determined data are
consistent and verifiable
Evidence that MCP’s use of codes to identify medical
events were correctly evaluated when classifying
members for inclusion or exclusion in the numerator
Evidence that MCP has counted each member and/or
event only once
Programming logic or demonstration that confirms that
any non-standard codes used in determining the
numerator have been mapped to a standard coding
scheme in a manner that is consistent, complete, and
reproducible
Programming logic or source code that identifies
process for integrating administrative and medical
record data for numerator
Programming logic and/or source code for arithmetic
calculation of each measure
Programming logic and/or source code for measures
with complex algorithms, to ensure adequate matching
and linkage among different types of data
Overall assessment: In the comments section, note
any recommendations to improve documentation or
demonstrations provided by the MCP
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 113

Worksheet 2.10. Measure Validation Findings
Instructions. For each performance measure, the EQRO can use this worksheet to document adherence to
guidance for (1) the denominator; (2) programming logic, source code, and calculations; (3) identifying medical
events; (4) time parameters; (5) exclusion criteria; (6) population estimates; (7) identifying the at-risk population; (8),
inclusion of qualifying medical events in the numerator; and (9) medical record data in the numerator. This worksheet
is relevant to Activity 2, Step 4, and Activity 3.

Audit Element
Denominator. For each performance measure, all
members of the relevant populations identified in the
performance measure specifications (who were
eligible to receive the specified services) were
included in the population from which the denominator
was produced. The eligible population included
members who received the services as well as those
who did not. The same standard applies to provider
groups or other relevant populations identified in the
specifications of each performance measure.
Programming logic, source code, and
calculations. For each measure, adequate
programming logic or source code identifies, tracks,
and links member enrollment within and across
product lines (e.g., Medicaid and CHIP), by age and
sex, as well as through possible periods of enrollment
and disenrollment) and appropriately identifies all
relevant members of the specified denominator
population for each of the performance measures.
This is determined by evaluating that:
1. Calculations of continuous enrollment criteria were
correctly carried out and applied to each measure
(if applicable)
2

Proper mathematical operations were used to
determine patient age or age range

3. The MCP can identify the variable(s) that define
the member’s sex in every file or algorithm needed
to calculate the performance measure
denominator, and the MCP can explain what
classification is carried out if neither of the
required codes is present
4. The MCP has correctly calculated member
months and member years, if applicable to the
performance measure
Identifying medical events. The MCP has properly
evaluated the completeness and accuracy of any
codes used to identify medical events, such as
diagnoses, procedures, or prescriptions, and these
codes have been appropriately identified and applied
as specified in each performance measure.

114 | PROTOCOL TWO

Met

Not
Met

N/A

Comments

Audit Element

Met

Not
Met

N/A

Comments

Time parameters. Any time parameters required by
the performance measure specification were followed
by the MCP (e.g., cut off dates for data collection,
counting 30 calendar days after discharge from a
hospital).
Exclusion criteria. Performance measure
specifications or definitions that exclude members
from a denominator were followed. (For example, if a
measure relates to receipt of a specific service, the
denominator may need to be adjusted to reflect
instances in which the patient refuses the service or
the service is contraindicated.)
Population estimates. Systems or methods used by
the MCP to estimate populations when they cannot be
accurately or completely counted (e.g., newborns) are
valid.
Identifying the at-risk population. The MCP has
used the appropriate data, including linked data from
separate data sets, to identify the entire at-risk
population.
Services provided outside the MCP. The MCP has
adopted and followed procedures to capture data for
those performance measures that could be easily
under-reported due to the availability of services
outside the MCP. (For some measures, particularly
those focused on women and children, the member
may have received the specified service outside of
the MCP provider base, such as children receiving
immunizations through public health services or
schools, access to family planning services. An extra
effort must be made to include these events in the
numerator.)
Inclusion of qualifying medical events. The MCP’s
use of codes to identify medical events (e.g.,
diagnoses, procedures, prescriptions) are complete,
accurate, and specific in correctly describing what
transpired and when. This included:
1, The MCP correctly evaluated medical event codes
when classifying members for inclusion or
exclusion in the numerator
2

The MCP avoided or eliminated all doublecounted members or numerator events

3. The MCP mapped any non-standard codes used
in determining the numerator in a manner that is
consistent, complete, and reproducible. The
EQRO assesses this through a review of the
programming logic or a demonstration of the
program

PROTOCOL TWO | 115

Audit Element

Met

Not
Met

N/A

Comments

4. Any time parameters required by the
specifications of the performance measure were
adhered to (i.e., that the measured event occurred
during the time period specified or defined in the
performance measure)
Medical record data. Medical record reviews and
abstractions were carried out in a manner that
facilitated the collection of complete, accurate, and
valid data by ensuring that:
1. Record review staff have been properly trained
and supervised for the task
2. Record abstraction tools required the appropriate
notation that the measured event occurred
3. Record abstraction tools required notation of the
results or findings of the measured event, if
applicable
4. Medical record data from electronic sources was
accurately extracted according to measure
specifications
5. Data included in the record extract files are
consistent with data found in the medical records
based on a review of a sample of medical record
for applicable performance measures
6. The process of integrating administrative data and
medical record data for the purpose of determining
the numerator is consistent and valid
Overall assessment: In the comments section, note
any recommendations (if applicable) to:
• Improve the denominator
• Improve programming logic, source code, or
calculations
• Improve the completeness or accuracy of the
codes used to identify medical events
• Improve the specified time parameters
• Improve adherence to the exclusion criteria
• Improve systems/methods to estimate populations
when they cannot be accurately counted
• Ensure all appropriate data are used to identify the
entire at-risk population
• Appropriately identify and include qualifying
medical events for the numerator
• Improve the proper collection of medical record
data extracted for inclusion in the numerator
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

116 | PROTOCOL TWO

Worksheet 2.11. Interview Guide for Assessing Processes and Procedures Used to
Produce Numerators and Denominators
Instructions. The following interview guide may be used to supplement findings reported in Worksheet 2.10 (as
described in Activity 2, Step 4). Please tailor the questions as appropriate.
MCP Contact and Background Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, email address, and date of interview, if applicable.
MCP name:
MCP contact name(s):
Title(s):
Mailing address:
Phone number(s):
Email address:
Interview Date:
EQRO reviewers:
Year of first Medicaid enrollment:
Year of first CHIP enrollment:
Year of first MCP performance report
(any product line)
1. If any part of your network/data/membership was excluded from a performance measure, how and why did you
decide to exclude it?
2. Why did you select the reporting methodology (e.g., administrative, or hybrid) used to create each of the
measures (where there was an option)?
3. Did you use the state technical specifications as the specifications for the programmers, or did your MCP write its
own instructions/translations for the programmers?
4. Are there any manual processes used for calculating denominators and/or numerators? Are manual processes
used for sampling?
5. Are any measures calculated by vendors? If yes, are they checked for accuracy? Please describe.
6. Do you have any concerns about the integrity of the information used to create any of the measures? Please
describe.
7. Do you know of any deviations from performance measure specifications that were necessary because of data
available or because of your MCP’s information system capabilities?
8. Other issues.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL TWO | 117

Worksheet 2.12. Policies, Procedures, and Data Used to Implement Sampling: Review
Checklist
Instructions. This checklist provides a list of documents, procedures, and data to assess the sampling process, if
applicable (as described in Activity 2, Step 5).

Documents

Reviewed

Not
Reviewed

Comments

Description of software used to execute sampling sort
of population files when sampling (e.g., systematic) is
used
Policies to maintain files from which the samples are
drawn in order to keep population intact in the event
that a sample must be re-drawn or replacements
made
Computer source code or logic identifying specified
sampling techniques, and documentation that the
logic matches the specifications set forth for each
performance measure, including sample size and
exclusion methodology
Methods used for sampling for measures calling for
hybrid data or medical record review
Documentation assuring that sampling methodology
treats all measures independently, and that there is
no correlation between drawn samples
Observation of or documentation of procedures in
which a biased sample was identified and corrected
Documentation of “frozen” or archived files from
which the samples were drawn, and if applicable,
documentation of the MCO’s process to re-draw a
sample or obtain necessary replacements
Overall assessment: In the comments section, note
any recommendations to improve documentation or
demonstrations to assess the sampling process
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

118 | PROTOCOL TWO

Worksheet 2.13. Sampling Validation Findings
Instructions. This checklist provides a list of documents, procedures, and data to assess the sampling process
across the following elements: (1) the MCP followed the specified sampling method to produce an unbiased sample
that is representative of the entire included population, (2) the MCP maintains its performance measurement
population files/data sets in a manner that allows a sample to be re-drawn or used as a source for replacement, (3)
sample sizes collected conform to the methodology set forth in the performance measure specifications and the
sample is representative of the entire population, and (4) for performance measures that include medical record
review (e.g., hybrid data collection methodology), proper substitution methodology was followed. This worksheet is
applicable to Activity 2, Step 4 and Activity 3.

1. Audit Element: Sampling method

Met

Not
Met

N/A

Comments

Met

Not
Met

N/A

Comments

Each relevant member or provider had an equal
chance of being selected; no one was systematically
excluded from the sampling.
The MCP followed the specifications set forth in the
performance measure regarding the treatment of
sample exclusions and replacements, and if any
activity took place involving replacements of or
exclusions from the sample, the MCP kept adequate
documentation of that activity
Each provider serving a given number of enrollees
had the same probability of being selected as any
other provider serving the same number of enrollees
The MCP examined its sampled files for bias, and if
any bias was detected, the MCP is able to provide
documentation that describes any efforts taken to
correct it
The sampling methodology employed treated all
measures independently, and there is no correlation
between drawn samples
Relevant members or providers who were not
included in the sample for the baseline measurement
had the same chance of being selected for the
follow-up measurement as providers who were
included in the baseline
Overall assessment: In the comments section, note
any recommendations to produce an unbiased,
representative sample

2. Audit Element: Performance measurement
files/data
The MCP has policies and procedures to maintain
files from which the samples are drawn in order to
keep the population intact in the event that a sample
must be re-drawn, or replacements made, and
documentation that the original population is intact

PROTOCOL TWO | 119

2. Audit Element: Performance measurement
files/data

Met

Not
Met

N/A

Comments

Met

Not
Met

N/A

Comments

Met

Not
Met

N/A

Comments

Overall assessment: In the comments section, note
any recommendations to improve file or data
maintenance

3. Audit Element: Performance measure
specifications
Sample sizes meet the requirements of the
performance measure specifications
The MCP has appropriately handled the
documentation and reporting of the measure if the
requested sample size exceeds the population size
The MCP properly oversampled in order to
accommodate potential exclusions
Overall assessment: In the comments section, note
any recommendations to improve adherence to
performance measure specifications

4. Audit Element: Medical record reviews
Substitution applied only to those members who met
the exclusion criteria specified in the performance
measure definitions or requirements
Substitutions were made for properly excluded
records and the percentage of substituted records
was documented
Overall assessment: In the comments section, note
any recommendations to improve use of proper
substitution methodology

120 | PROTOCOL TWO

Worksheet 2.14. Framework for Summarizing Information About Performance Measures
Instructions. Use this worksheet or a similar tool to summarize the results for each performance measure validated
for each managed care plan. This worksheet can be used as a framework for summarizing validation at the plan
level. In addition, the information in this worksheet can be aggregated across plans and measures to generate
information on state-level performance and areas for improvement.

1. Overview of Performance Measure
Managed Care Plan (MCP) name:
Performance measure name:
Measure steward:
Agency for Healthcare Research and Quality (AHRQ)
Centers for Disease Control and Prevention (CDC)
Centers for Medicare & Medicaid Services (CMS)
National Committee for Quality Assurance (NCQA)
The Joint Commission (TJC)
No measure steward, developed by state/EQRO
Other measure steward (specify) _____________________________________________
Is the performance measure part of an existing measure set? (check all that apply)
HEDIS®
CMS Child or Adult Core Set
Other (specify) ____________________________________________
What data source(s) was used to calculate the measure? (check all that apply)
Administrative data (describe) ________________________________
Medical records (describe) __________________________________
Other (specify) ____________________________________________
If the hybrid method was used, describe the sampling approach used to select the medical records:
Not applicable (hybrid method not used)
Definition of denominator (describe):
Definition of numerator (describe):
Program(s) included in the measure:
CHIP

Medicaid (Title XIX) only

CHIP (Title XXI) only

Medicaid and

Measurement period (start/end date)

PROTOCOL TWO | 121

Performance Measure Results (If measure contains more than one rate, add columns to the
table)

Performance
measure

Rate 1

Rate 2

Rate 3

Rate 4

Numerator
Denominator
Rate

Describe any findings from the ISCA or other information systems audit that affected the reliability or validity of the
performance measure results.
Not applicable (ISCA not reviewed)
Describe any findings from medical record review that affected the reliability or validity of the performance measure
results.
Not applicable (medical record review not conducted)
Describe any other validation findings that affected the accuracy of the performance measure calculation.

Validation rating:

High confidence

Moderate confidence

Low confidence

No confidence

122 | PROTOCOL TWO

Example of Worksheet 2.14. Framework for Summarizing Information about Performance
Measures
Instructions. Use this worksheet or a similar tool to summarize the results for each performance measure validated
for each managed care plan. This worksheet can be used as a framework for summarizing validation at the plan
level. In addition, the information in this worksheet can be aggregated across plans and measures to generate
information on state-level performance and areas for improvement.

1. Overview of Performance Measure
Managed Care Plan (MCP) name: Plan A
Performance measure name: Follow-Up After Hospitalization for Mental Illness: 6-20 (FUH-CH)
Measure steward:
Agency for Healthcare Research and Quality (AHRQ)
Centers for Disease Control and Prevention (CDC)
Centers for Medicare & Medicaid Services (CMS)
National Committee for Quality Assurance (NCQA)
The Joint Commission (TJC)
No measure steward, developed by state/EQRO
Other measure steward (specify) _____________________________________________
Is the performance measure part of an existing measure set? (check all that apply)
HEDIS®
CMS Child or Adult Core Set
Other (specify) ____________________________________________
What data source(s) was used to calculate the measure? (check all that apply)
Administrative data (describe): The administrative data source is the state’s MMIS and data submitted by the
managed care plans.
Medical records (describe) __________________________________
Other (specify) ____________________________________________
If the hybrid method was used, describe the sampling approach used to select the medical records:
Not applicable (hybrid method not used)
Definition of denominator (describe): Medicaid rates include managed care population (4 MCOs) age 6 and older.
Definition of numerator (describe):
•

7-day follow-up: A follow-up visit with a mental health practitioner within 7 days after discharge. This
includes visits that occur on the date of discharge.

•

30-day follow-up: A follow-up visit with a mental health practitioner within 30 days after discharge. This
includes visits that occur on the date of discharge.

Program(s) included in the measure:
CHIP

Medicaid (Title XIX) only

CHIP (Title XXI) only

Medicaid and

Measurement period (start/end date): January 1, 2020 to December 31, 2020.

PROTOCOL TWO | 123

2. Performance Measure Results (If measure contains more than one rate, add columns to the
table)
Performance
measure

Rate 1

Rate 2

Numerator

6,723

8,476

Denominator

12,007

Rate

56.0

70.6

(7-day follow-up)

(30-day follow-up)

Rate 3

Rate 4

3. Performance Measure Validation Status
Describe any deviations from the technical specifications and explain reasons for deviations (such as deviations in
denominator, numerator, data source, measurement period, or other aspect of the measure calculation).
Plan A was compliant with the HEDIS® Information System Standards and HEDIS® Determination Standards, and
continues to use NCQA-certified software vendors for HEDIS® measure production.
Describe any findings from the ISCA or other information systems audit that affected the reliability or validity of the
performance measure results.
Not applicable (ISCA not reviewed)
Describe any findings from medical record review that affected the reliability or validity of the performance measure
results.
Not applicable (medical record review not conducted)
Describe any other validation findings that affected the accuracy of the performance measure calculation.
No findings to report.

Validation rating:

High confidence

Moderate confidence

Low confidence

No confidence

“Validation rating” refers to the EQRO’s overall confidence that the calculation of the performance measure adhered
to acceptable methodology.
EQRO recommendations for improvement of performance measure calculation:
The FUH-CH measure represents one of the objectives in the state’s Quality Strategy (e.g., child health, prevention,
and screening services), which seeks to assure timely, high-quality health care for all [State Medicaid Program
Name] members. The EQRO has no recommendations to improve the performance measure calculation.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

END OF WORKSHEETS FOR PROTOCOL 2

124 | PROTOCOL TWO

Protocol 3. Review of Compliance With
Medicaid and CHIP Managed Care
Regulations

ACTIVITY 1: ESTABLISH COMPLIANCE THRESHOLDS
ACTIVITY 2: PERFORM THE PRELIMINARY REVIEW (PRE-SITE VISIT)
ACTIVITY 3: CONDUCT MCP SITE VISIT
ACTIVITY 4: COMPILE AND ANALYZE FINDINGS (POST-SITE VISIT)
ACTIVITY 5: REPORT RESULTS TO THE STATE

Background
This protocol is used to determine the extent to which Medicaid and CHIP
managed care plans (MCPs) are in compliance with federal standards. The
Centers for Medicare & Medicaid Services (CMS) developed standards for
MCPs, which are codified at 42 CFR 438 and 42 CFR 457, as revised by the
Medicaid and CHIP managed care final rule issued in 2016. As noted in the
Introduction, states have the option to use information from a Medicare or
private accreditation review of an MCP to provide information for the annual
EQR instead of conducting this mandatory EQR-related activity. 52, 53

If the state elects to use nonduplication for this mandatory EQR-related activity (42 CFR 438.360,
Nonduplication of mandatory activities with Medicare or accreditation review), then the state must
ensure that all information from the Medicare or private accreditation review is provided to the EQRO
for analysis and inclusion in the annual EQR technical report. (See 42 CFR 438.360(a)(1)–(3) for
additional details regarding the circumstance under which nonduplication is an option). Use of
nonduplication must be identified in the state’s quality strategy (see 42 CFR 438.360(c) and
438.340(b)(10)). Any requirements in this protocol which are not addressed via the review used for
nonduplication must still be addressed through this protocol. CHIP cross-references to this requirement
at 457.1250, but does not allow for the use of Medicare review activities for the purposes of
nonduplication.

A state may not utilize nonduplication if Medicare has accepted only an attestation of a plan’s QIP. In
the context of this EQR-related activity, the QIP would have to undergo validation as part of a Medicare
review in order for nonduplication to be an option. See 42 CFR 438.360(a)(2).

PROTOCOL THREE

A MANDATORY EQR-RELATED ACTIVITY

125

Regulations Subject to Compliance Review
The standards that are the subject of this protocol are contained in 42 CFR 438, part 56, 100,
114, Subparts D and QAPI. 54 The scope of those sections includes: 55
•

Disenrollment requirements and limitations 438.56

•

Enrollee rights requirements 438.100

•

Emergency and post-stabilization services 438.114

•

Availability of services 438.206

•

Assurances of adequate capacity and services 438.207

•

Coordination and continuity of care 438.208

•

Coverage and authorization of services 438.210

•

Provider selection 438.214

•

Confidentiality 438.224

•

Grievance and appeal systems 438.228

•

Subcontractual relationships and delegation 438.230

•

Practice guidelines 438.236

•

Health information systems 438.242

•

Quality assessment and performance improvement program 438.330

Standards Subject to this
Protocol Added from the 2020
Final Rule

•

Disenrollment requirements and
limitations 438.56

•

Enrollee rights requirements
438.100

•

Emergency and post-stabilization
services 438.114

Additional Areas for Potential Compliance Review
CMS encourages states to consider expanding the scope of the review to cover compliance with
federal and state requirements beyond those specified in 42 CFR 438, including other state
statutory, regulatory, or contractual requirements related to the following areas, if applicable: 56
•

Accessibility, including physical accessibility of service sites and medical and diagnostic
equipment; accessibility of information (compliance with web-based information, literacy
levels of written materials, and alternate formats); and other accommodations. See Section
508 of the Rehabilitation Act [29 U.S.C. 794d])

CHIP cross-references to these requirements at 42 CFR 457.1230, 457.1233, and 457.1240, except as noted. For more
information, see https://www.federalregister.gov/documents/2016/05/06/2016-09581/medicaid-and-childrens-healthinsurance-program-chip-programs-medicaid-managed-care-chip-delivered#sectno-citation-%E2%80%89438.206.

Certain requirements in Subparts A, B, C, and F are incorporated into the compliance review through interaction with Subparts
D and E.

For more information, see the CMS MLTSS EQR guidance document at
https://www.medicaid.gov/Medicaid/downloads/cmcs-eqr-protocols.pdf.

126 | PROTOCOL THREE

•

Availability and use of home- and community-based services (HCBS) as alternatives to
institutional care, so individuals can receive the services they need in the most integrated
setting appropriate

•

Credentialing or other selection processes for long-term services and supports (LTSS)
providers, including those required where the enrollee can choose their caregiver (such as
verification of completion of criminal background checks)

•

Person-centered assessment, person-centered care planning, service planning and
authorization, service coordination and care management for LTSS, including
authorization/utilization management for LTSS and any beneficiary rights or protections
related to care planning and service planning such as conflict-free case management, selfdirection of services, and appeal rights related to person-centered planning

•

Integration of managed medical, behavioral, and LTSS

Frequency of Compliance Review and Manner of Reporting
Federal regulations require MCPs to have undergone a review within the three-year period
preceding each annual EQR to determine MCP compliance with federal standards as
implemented by the state. States may choose to direct their EQROs to review all applicable
standards at once or may spread the review over a three-year cycle in any manner they choose
(for example, fully reviewing a third of plans each year or conducting a third of the review on all
plans each year). However, if an EQR technical report summarizes a compliance review that
does not include all required components, the report should clearly describe:
1. The three-year period covered by the current compliance review cycle
2. The quality standards not included in the current report
3. A summary of findings from all previous reviews within the current review cycle
4. The state’s schedule for review of the remaining standards

Getting Started on Protocol 3
This protocol describes the process EQROs may use to determine MCP compliance with federal
Medicaid and CHIP managed care regulations. In general, the EQRO must:
1. Review program documents and conduct interviews with MCP personnel to collect
information, and
2. Analyze information collected and make compliance determinations
To complete this protocol, the EQRO must undertake five activities for each MCP (Figure 3.1).

PROTOCOL THREE | 127

Figure 3.1. Protocol 3 Activities

One supplemental resource is available to help EQROs conduct the compliance review:
•

Worksheets for Protocol 3. Compliance Review Tools, which can be used to (1) structure and
conduct the review of MCP documentation to determine compliance with the applicable
federal regulatory and/or state provisions; (2) score MCPs’ compliance with federal and state

128 | PROTOCOL THREE

regulations; (3) develop site visit agendas, 57 and (4) guide compliance interviews of MCP
staff
The remainder of this protocol outlines the steps associated with Activities 1 through 5.

Activity 1: Establish Compliance Thresholds
In this activity, the EQRO works with the state to define levels of compliance for use throughout
the compliance review. Activity 1 includes two steps:
1. Determine how the state has implemented federal quality standards
2. Define the benchmark for compliance against which the MCP will be measured or scored

Step 1: Collect Information from the State
Some regulatory provisions allow the state to establish more
stringent standards for their MCPs than are mandated by
federal regulation. Additional state requirements may be
found in MCP contracts, state managed care quality
strategies, state statutes and regulations, or other resources.
States cannot establish standards that are less stringent than
federal regulation, nor can states exempt MCPs from
compliance standards. Therefore, to complete the
compliance assessment, the EQRO will need to know the
state’s requirements for its MCPs in the context of each
required federal standard. This documentation may be
provided to the EQRO in hard copy, digital copy, or both
formats. Worksheet 3.1 lists the types of documents that the
state may provide to the EQRO about state standards and is
organized according to federal regulatory provision.

WORKSHEET 3.1
Resource for Activity 1, Step 1
Worksheet 3.1. Compliance Review
• Includes a list of the types of state
policy/regulation information needed
to determine MCP compliance, as
well as documents the MCP may
provide the EQRO to demonstrate
the MCP's compliance with federal
regulations and state standards (see
Activity 2, Step 2)
• The completed Compliance Review
Worksheet is a primary data source
for analyses and a comprehensive
record of compliance protocol EQRrelated activities

In the event that onsite activities are not feasible due to the COVID-19 pandemic, site visits may be conducted virtually.

PROTOCOL THREE | 129

Step 2: Define Benchmarks for Compliance
EQRO determinations will be based on compliance
definitions set in advance by the state for each federal
and/or state regulatory provision, component of a provision,
and/or requirement or standard based on its expectations of
MCP performance. While states may define degrees of
compliance with each benchmarked standard, a definition
for full compliance must be clearly understood by the
EQRO and MCP before the review.

Activity 2: Perform the Preliminary
Review (Pre-Site Visit)

WORKSHEET 3.2
Resource for Activity 2, Step 2
Worksheet 3.2. Compliance Definitions
• Provides compliance definitions and
examples of compliance rating scales
○ Compliance rating scales may be
adjusted to best suit a state’s
needs

Site visits are an effective way to collect the information needed for quality oversight and
compliance determination. However, they require careful planning to maximize the information
obtained and to minimize the time required for collecting that information. This activity should
begin from 2 to 6 months in advance of the planned visit.
EQROs, MCPs, and states that opt to conduct virtual compliance visits should plan those visits
with the same disciplined approach they would for an onsite visit as described in this protocol.

Step 1: Establish Early Contact with the MCP
It is important for the EQRO to establish and maintain consistent communication with the MCP
throughout the compliance review. The EQRO is responsible for developing a communication
plan specifying expectations for all parties involved. The EQRO should establish a single point
of contact with the MCP, and in turn, the MCP representative can organize the response from the
MCP and determine which additional staff members should be involved during the review.

130 | PROTOCOL THREE

Step 2: Perform a Document Review
The purpose of the document review is to identify gaps in
information to ensure a comprehensive EQR process and
productive interactions with the MCP during the site visit.
Before planning a site visit, the EQRO should gather and
assess as much information about the MCP and its practices
as possible. Document review includes gathering
information about the MCP’s background, including its
structure, enrolled population, providers, services, resources,
locations, delegated functions and services, and contractors.
Some information may be available from the state, while
some may be obtained from the MCP. The following list
suggests the type of information that would be useful during
a preliminary document review:

WORKSHEET 3.1
Resource for Activity 2, Step 2
Worksheet 3.1. Compliance Review
• Includes a list of the types of state
policy or regulatory information
needed to determine MCP
compliance (see Activity 1, Step 1),
as well as documents the MCP may
provide the EQRO to demonstrate
the MCP's compliance with federal
regulations and state standards
• The completed Compliance Review
Worksheet is a primary data source
for analyses and a comprehensive
record of compliance protocol EQRrelated activities

•

Organization name and mailing address

•

Contact person’s name, title, phone number, and e-mail
address

•

Site visit location

•

Organizational charts or other descriptions of the MCP

•

Product lines offered

•

Total individuals enrolled in the current and previous year

•

Total number of network practitioners in the current and previous year, with a breakdown by
type (such as primary care, OB/GYN, and other specialties)

•

Total number of network organizational providers (hospitals, ambulatory care, home care,
laboratories, etc.)

•

Service descriptions and benefit designs available to enrollees

•

Delegated activities, including MCP subcontractors

The EQRO should inform the MCP of any missing information before the site visit, to allow the
MCP to respond in a timely manner (either by providing the documentation to the EQRO prior to
the site visit for review or ensuring its availability during the site visit document review; see
Activity 3, Step 5). The EQRO should maintain consistent documentation by recording

PROTOCOL THREE | 131

preliminary document review findings or questions for follow-up during the site visit in
Worksheet 3.1. 58

Activity 3: Conduct MCP Site Visit
The purpose of the MCP site visit is to collect the information necessary to assess the MCP’s
compliance with federal and state regulations through additional document review and
interviews. The EQRO should plan the site visit, whether onsite or virtual, in accordance with the
compliance review plan established in Activity 1 of this protocol. As noted in Activity 2, the
EQRO should review MCP policies and procedures before the site visit to expedite the process.
Steps 1 through 5 provide guidance in preparing for the site visit.

Step 1: Determine Site Visit Length and Dates
The length of a comprehensive site visit will vary according to the scope of the review, the
complexity of the organization being reviewed, the number of reviewers available to conduct the
review, and the amount of information collected before the site visit. A typical site visit requires
3 to 5 days. To schedule the site visit, the EQRO should offer the MCP contact a range of dates
to determine when essential staff are available.

Step 2: Identify the Number and Types of Reviewers Needed
Reviewers should be skilled interviewers with the ability to read and process a variety of data in
order to determine whether an MCP is in compliance with the regulations. Knowledge or
experience in state Medicaid and CHIP programs and managed care is highly desirable.
Reviewer orientation and training should be held to ensure familiarity with the regulatory
provisions, the evaluation process, and performance expectations.
The number of reviewers needed to conduct the assessment should be based on the
characteristics of the MCP being evaluated. Consideration should be given to the size and
complexity of the MCP, including the size of the provider network, number of enrollees, and the
scope of programs in the state contract. If multiple reviewers are participating in the site visit(s),
the EQRO should identify in advance each reviewer’s responsibility for assessing specific
standards, reviewing specific documents, and conducting interviews.

In addition to the document review described here as part of the preliminary document review, the previous version of this
protocol included a step in Activity 3 for document review while onsite. However, due to the widespread use of digital
documentation, an additional document review during the site visit (onsite or virtual) may not be needed. If a document review
is determined by the EQRO to be necessary (and negotiated with the MCP), the EQRO may conduct it during the site visit.
That step would occur before Activity 3, Step 5.

132 | PROTOCOL THREE

Step 3: Develop a Site Visit Agenda
Clear expectations are essential for an efficient and
effective site visit. An agenda sets the expectations and
schedule for the review. It also assists both the MCP and
EQRO in planning staff participation, gathering
documentation, and finalizing logistics such as arranging
locations for document review and interviews. The EQRO
should consult with the MCP throughout the agenda setting
process to ensure the inclusion of appropriate staff and
subcontractors.

Step 4: Provide Preparation Instructions and
Guidance to the MCP

WORKSHEET 3.3
Resource for Activity 3, Step 3
Worksheet 3.3. Sample Site Visit
Agenda
• This template can be used to develop
the site visit agenda. It is intended to
help the MCP and EQRO in planning
staff interviews, gathering
documents, and finalizing logistics

The EQRO should send clear instructions and guidance to the MCP before the site visit. In
preparation for the site visit, the EQRO should provide MCPs with the following information:
1. The scope of the assessment
2. How the review will be conducted
3. List of required documents
4. Instructions for how documents for review should be organized
5. Forms or other data gathering instruments that should be completed before arrival (such as
the Information Systems Capability Assessment (ISCA); see Appendix A)
6. Reports from prior reviews and subsequent MCP corrective actions
7. Names and contact information for expected interview participants, and
8. Administrative needs of the reviewers

PROTOCOL THREE | 133

Step 5: Conduct MCP Interviews
The purpose of MCP interviews is to collect data to
supplement and verify what is learned through document
review. In preparation for the site visit, the EQRO should
review the standards identified in the state documents
obtained in Activity 1 and the findings from Activity 2,
Step 2. During the site visit, MCP staff should be available
if the EQRO has questions or difficulty locating any
needed additional documents or other information. The
EQRO should notify the MCP during the site visit of any
missing information to allow the MCP to respond in a
timely manner. The EQRO should maintain consistent
documentation by adding to Worksheet 3.1 any findings
based on any additional information or documents
provided by the MCP.
Prepare for the Interviews
Interviews should be tailored to the MCP being evaluated
and the role of the interviewee. When planning for the
interview, the EQRO should:
•

Prepare a list of issues to be addressed in each
interview, based on federal regulatory provisions, state
standards, MCP organization characteristics, and other
information gathered during pre-site visit document
reviews

WORKSHEET 3.1
WORKSHEET 3.4
Resource for Activity 3, Step 5
Worksheet 3.1. Compliance Review
• Includes a list of the types of
documents the MCP may provide the
EQRO to demonstrate the MCP’s
compliance with federal regulations
and state standards.
• Provides space for reviewers to
document follow-up to questions from
the pre-site visit documentation
review. The completed Compliance
Review Worksheet is a primary data
source for analyses and a
comprehensive record of compliance
protocol EQR-related activities
Worksheet 3.4. Compliance Interview
Questions
• Provides questions that are intended
to guide the reviewer's discussion
with MCP staff to help determine
compliance with state and federal
requirements
• The questions are first organized by
MCP staff roles and then by
regulatory provision

•

Review the MCP’s anticipated interview participants,
and identify topics that will promote an inclusive
discussion

•

If multiple reviewers are assigned to an MCP, assign primary roles to each reviewer (such as
interviewer or note-taker), while allowing for shared roles and responsibilities as appropriate
throughout the site visit

It is strongly recommended that the EQRO completes the document review (Activity 2, Step 2)
before the interviews. Some interview participants may provide additional documents during
their interview. This might be done when such documents are vital to the discussion or if the
review of the documents will benefit from joint review by all participants.

134 | PROTOCOL THREE

Interview Participants
Interviews should be conducted with groups, rather than with single individuals, because rarely
does one individual have sole responsibility for a particular function. Interview groups should
include participants that represent different functions, services, or departments of the MCP to
enable the EQRO to collect multiple perspectives about an issue. Group interviews are also an
opportunity for MCP staff to learn about compliance activities in other departments. The EQRO
has the discretion to meet with less than the full list of MCP-recommended employees in
situations where the EQRO feels that it can obtain the required information without the
attendance of all MCP employees listed in the protocol, or the MCP has identified a more
appropriate person to address questions but is not on the recommended list. Worksheet 3.4.
includes questions for the following groups:
•

MCP leaders

•

MCP information systems staff

•

Quality assessment and performance improvement program staff

•

Provider/contractor services staff

•

Enrollee services staff, including grievance and appeal staff

•

Utilization management staff

•

Medical director(s)

•

Case managers and care coordinators, and

•

MCP providers and contractors, as appropriate and as time and resources permit

Interview Process
The EQRO should provide the MCP interview participants with an interview agenda before the
interview, which includes the interview goals, issues, topics, and a list of related materials or
documents. Effective facilitation of an interview with an individual or a group requires that the
EQRO:
•

Maintain control of the interview discussion by politely redirecting participants to the topic
or question as necessary

•

Adhere to the time frames outlined in the agenda

•

Listen carefully to participants and summarize or restate participant responses to ensure
understanding

•

Take notes using the Worksheet 3.1 or similar tool, or according to the Compliance Review
Questions provided in Worksheet 3.4

•

Review documents provided during the interview at an appropriate time based on the content
and purpose of sharing the document
PROTOCOL THREE | 135

•

Conclude the interview with a review of the outlined goals and compliance levels to ensure
an understanding of the extent to which they were met, and

•

Provide information about next steps as appropriate

Interviews & Systems Capabilities
States have the opportunity to expand the roles of other state agencies in terms of their
responsibilities related to data exchanges, EHRs, interoperability, care coordination, and
Medicaid or CHIP waivers. At the state’s discretion, it may determine:
•

Whether the EQRO will review the state’s health information technology (HIT) plan for
HITECH and meaningful use with respect to validation of performance measures or
performance improvement project activities, and

•

How the MCP’s systems will support state efforts in a valid way

The EQRO is required to conduct an Information Systems Capability Assessment (ISCA) to
support the compliance review under Protocol 3. Box 3.1 lists resources to conduct an ISCA.
Box 3.1. Resources to Conduct an ISCA
The ISCA is used to validate MCP information systems, processes, and data. The ISCA provides a foundation for the
EQR-related activities.
• Appendix A explains how to conduct the ISCA.
• Worksheet A.1. ISCA Tool is completed by the MCP and documents the capabilities of the information systems,
processes, and data
• Worksheet A.2. ISCA Interview Guide is used by EQROs to conduct follow-up interviews with staff to record
responses and document specific issues based on findings from Worksheet A.1

Step 6: Conduct Exit MCP Interviews
The EQRO conducts an exit interview at the conclusion of the site visit with MCP staff. The
purpose of the exit interview is to clarify the EQRO’s understanding of the information collected
throughout the compliance review process. The EQRO should provide the MCP with the
opportunity to respond to initial compliance issues to ensure the findings are due to true noncompliance and not due to misunderstanding or misinterpretation of MCP documents and
interviews.

Activity 4: Compile and Analyze Findings (Post-Site Visit)
Post site-visit activities include (1) collecting and documenting additional information as needed,
and (2) analyzing data compiled pre-, during, and post-site visit to make compliance
determinations for each regulatory provision.

136 | PROTOCOL THREE

Step 1: Collect Supplemental Information
In addition to information collected during the site visit, the EQRO should consider other sources
of information that confirm the MCP’s compliance with federal regulations and state standards.
Additional sources should include the following:
•

Results of Medicaid and CHIP beneficiary surveys (see Protocol 6 about administering
surveys)

•

Results of independent assessments of the MCP’s information systems (see Appendix A
about performing an ISCA)

•

Results of independent assessments of MCP encounter data (see Protocol 5 about validating
encounter data)

•

Results of independent validations of MCP performance measures (see Protocol 2 and
Protocol 7 for validating and calculating performance measures, respectively)

•

Results of independent validation of performance improvement projects (PIPs) (see Protocol
1 and Protocol 8 about validating and implementing PIPs, respectively or Protocol 9 about
conducting a Focus Study)

•

Additional materials requested during or after the site visit, such as grievance and appeal
reports and analyses

Step 2: Compile Data and Information
EQROs should use Worksheet 3.1 (or a similar template)
to document additional information they review, including
sources of the information and their findings about the
MCP’s compliance. EQROs should refer to Worksheet 3.2
for standardized compliance definitions and examples of
rating scales.

Step 3: Analyze Findings
One commonly used approach to analyzing EQR findings
is to assign a numerical value to indicate the degree of
compliance with a given regulatory provision. Using the
resources in Worksheet 3.1, the EQRO should document
both a compliance score for each regulatory provision, as
well as details and justification for the compliance
determination.

WORKSHEET 3.1
WORKSHEET 3.2
Resource for Activity 4, Step 2
Worksheet 3.1. Compliance Review
• Includes a list of the types of
documents the MCP may provide the
EQRO to demonstrate the MCP's
compliance with federal regulations
and state standards
• Can be used by the EQRO to
document compliance scores and
justification
Worksheet 3.2. Compliance Definitions
• Provides compliance definitions and
examples of compliance rating scales
• Compliance rating scales may be
adjusted to best suit a state’s needs

PROTOCOL THREE | 137

Regardless of the number of points on a scale, each level
Tips for analyzing EQR
of compliance must be defined clearly for the state, the
compliance findings
EQRO, and the MCP before beginning the review. While
The EQRO is encouraged to use a
one scale may serve as the primary method of assigning
numeric score to indicate the degree
levels of compliance, it does not preclude the combined
of compliance with each regulatory
use of another scale. For example, a five-point compliance
provision and to summarize the
overall percentage of elements scored
scale may be appropriate for most of the provisions, but
as compliant. The EQRO should
some provisions may be dichotomous (e.g., met or not
clearly document the scoring methods
met). When determinations are made for levels of
and evidence used to determine
compliance.
compliance other than ‘met’ or ‘not met,’ such as
‘partially met,’ the EQRO should clearly identify specific
deficiencies, as well as the rationale for and evidence of the deficiency.

Activity 5: Report Results to the State
Step 1: Submit a Report Outline to the State
The EQRO should develop a report outline and submit it to the state for approval. The outline
will then be used by the EQRO to draft a report to the state with the results of the MCP’s
compliance with federal and state requirements.

Step 2: Submit a Final Determination Report to the State
Because the state may use the report to meet its reporting requirements for federal or state
agencies, the state legislature, local advocacy groups, and other interested parties, the state may
need certain types of information presented in a specific format. While non-summarized findings
might be of interest to some individuals, the report should include an overall summary of
findings for compliance with regulatory provisions.
By design, Worksheet 3.1 separates the regulatory provisions into three major sections:
1. Standards, including enrollee rights and protections
2. Quality assessment and performance improvement (QAPI) program
3. Grievance system

Step 3: Submit Other Reports Requested by the State
The state may request a specific format for reporting results back to the MCP. Some options for
reporting evaluation results to the MCP include:
1. Compliance Issues Only. Reviewers provide verbal feedback about general compliance
issues they have identified during the course of conducting the compliance review EQRrelated activity. Neither compliance determinations for individual regulatory provisions nor
findings for a level of MCP performance are discussed. This type of feedback typically is
138 | PROTOCOL THREE

provided to the MCP leadership during a closing session or exit interview at the site visit.
This provides the MCP the opportunity to offer additional information if evidence of
compliance is available
2. Compliance Issues Specific to Regulatory Provisions. Reviewers provide verbal feedback
for regulatory provisions or components of provisions that are determined less than fully
compliant, in accordance with the compliance thresholds established by the state before the
review. Findings for a level of MCP performance are not discussed. This type of feedback is
typically presented to the MCP leadership during a closing session or exit interview at the
site visit. This provides the MCP the opportunity to offer additional information if evidence
of compliance is available
3. Compliance Determinations and Deficiency Report. Reviewers provide verbal and/or
written feedback about identified compliance issues, compliance ratings for regulatory
provisions, and an overall finding for MCP performance, highlighting areas of deficiency that
will be presented to the state
END OF PROTOCOL 3

PROTOCOL THREE | 139

Worksheets for Protocol 3: Compliance Review Tools
Instructions. Use these or similar worksheets to assess the MCP’s compliance with federal regulations and state
standards. These worksheets include a tool for document review, compliance definitions, a sample site visit agenda,
and compliance review interview questions. Each worksheet can be adapted as needed. This tool includes the
following worksheets crosswalked to the applicable Activity and Step:
Worksheet Name
Worksheet 3.1. Compliance Review

Protocol Activity and Step
Activity 1. Step 1. Collect Information from the State
Activity 2. Step 2. Perform a Document Review
Activity 3. Step 5. Conduct MCP Interviews
Activity 4. Step 2. Compile Data and Information
Activity 4. Step 3. Analyze Findings
Activity 5. Step 2. Submit a Final Determination Report

Worksheet 3.2 Compliance Definitions

Activity 1. Step 2. Define Levels of Compliance
Activity 4. Step 3. Analyze Findings

Worksheet 3.3. Sample Site Visit Agenda

Activity 3. Step 3. Develop a Site Visit Agenda

Worksheet 3.4. Compliance Review Interview Questions Activity 3. Step 5. Conduct MCP Interviews

140 | PROTOCOL THREE

Worksheet 3.1. Compliance Review
Instructions. Worksheet 3.1 includes a list of the types of documents the MCP may provide to the EQRO to
demonstrate the MCP’s compliance with federal regulations and state standards. It separates the regulatory
provisions into three major sections:
1. Standards, including enrollee rights and protections
2. Quality assessment and performance improvement (QAPI) program
3. Grievance system
This template may be used to track which MCP documents can provide the rationale for the compliance
determination for each regulatory provision (or component). This completed worksheet is intended to record
compliance activities to support the analyses.
Note: In the template, MCP documents are identified using generic names, except in instances where the regulatory
provisions refer to and require a specific document be present and reviewed for content.
The subject matter of each example MCP document is indicated in parenthesis as follows:
AM = Administrative/ Managerial
PS = Provider/Contractor Services
UM = Utilization Management
ES = Enrollee Services
IS = Information Systems
SP = Staff Planning, Education, Development and Evaluation
The subject matter designation does not imply that the document cannot be used as a data source for addressing
other provision issues, or that it should be the sole source of data in evaluating compliance with the provisions noted.
Refer to Worksheet 3.2, Compliance Definitions for more information on approaches to compliance scoring.

PROTOCOL THREE | 141

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

Availability of
services

• Service planning documents and
• The state’s provider-specific network
provider network planning documents
adequacy requirements and standards
(e.g., geographic assessments,
(and exceptions, if any)
Medicaid: 42 CFR
provider network assessments,
438.206 (availability of • The state’s requirements for the MCP
enrollee demographic studies,
provider
directory
services) and 42 CFR
population needs assessments)(AM)
• Information on the documentation that
10(h) provider
• Service availability and accessibility
the state uses to support its
directory)
expectations and standards (AM)
certification that the MCP complied
CHIP: 42 CFR
• Other performance standards and
with the state’s requirements for
457.1230(a)
quality indicators established by the
availability and accessibility of
services, including the adequacy of
the provider network

MCP (AM)

(See Worksheet
3.2 for
approaches to
compliance
scoring)
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

CHIP:
• Any measurement or analysis reports
Fully Met
on service availability and
accessibility (AM)
Substantially Met
• List of all care and service providers
in the MCP’s network (may be the
same as the provider directory) (AM)
• Organization strategic plans (AM)
• Administrative policies and
procedures (AM)

Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Medicaid and CHIP and other
enrollee survey results (AM)
• Utilization management policies and
procedures (UM)
• Service authorization policies and
procedures (UM)
• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)
• Provider/Contractor oversight and
evaluation policies and procedures,
audit tools (PS)
• Medicaid and CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid and CHIP enrollee
handbooks (ES)
• Medicaid and CHIP provider directory
• Medicaid and CHIP enrollee
orientation curriculum (ES)
• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)
Furnishing of
services and timely
access

• Obtain a copy of the state Medicaid
and CHIP agency’s standards for
timely enrollee access to care and
services required of Medicaid and
CHIP and MCPs.

Medicaid:
• Service planning documents and
provider network planning documents Fully Met
(e.g., geographic assessments,
Substantially Met
provider network assessments,

PROTOCOL THREE | 142

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

Medicaid: 42 CFR
438.206(c)(1):
Furnishing of services
and timely access

• Service availability and accessibility
expectations and standards (AM)

CHIP: 42 CFR
457.1230(a):
Availability of services

• Other performance standards and
quality indicators established by the
MCP (AM)

enrollee demographic studies,
population needs assessments)(AM)

(See Worksheet
3.2 for
approaches to
compliance
scoring)
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

CHIP:
• Any measurement or analysis reports Fully Met
on service availability and
Substantially Met
accessibility (AM)
Partially Met
• List of all care and service providers
in the MCP’s network (may be the
Minimally Met
same as the provider directory) (AM)
Not Met
• Organization strategic plans (AM)
Not Applicable
• Administrative policies and
Reviewer Notes:
procedures (AM)
• Medicaid and CHIP and other
enrollee survey results (AM)
• Utilization management policies and
procedures (UM)
• Service authorization policies and
procedures (UM)
• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)
• Provider/Contractor oversight and
evaluation policies and procedures,
audit tools (PS)
• Medicaid and CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid and CHIP Enrollee
Handbooks (ES)
• Medicaid and CHIP provider directory
• Medicaid and CHIP Enrollee
Orientation Curriculum (ES)
• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Access and cultural
considerations
Medicaid: 42 CFR
438.206(c)(2):
Furnishing of services
and cultural
considerations.

• Descriptive information on the state’s
efforts to promote the delivery of
services in a culturally competent
manner to all enrollees, including
those with limited English proficiency
and diverse cultural and ethnic
backgrounds.
• The requirements the state has
communicated to the MCP with
respect to how the MCP is expected to

• Service planning documents and
provider network planning documents
(e.g., geographic assessments,
provider network assessments,
enrollee demographic studies,
population needs assessments)(AM)
• Service availability and accessibility
expectations and standards (AM)

Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

PROTOCOL THREE | 143

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

CHIP: 42 CFR
457.1230(a): Access
standards

participate in the state’s efforts to
promote the delivery of services in a
culturally competent manner.

Applicable MCP documents
• Other performance standards and
quality indicators established by the
MCP (AM)

(See Worksheet
3.2 for
approaches to
compliance
scoring)
CHIP:
Fully Met

Substantially Met
• Any measurement or analysis reports
Partially Met
on service availability and
accessibility (AM)
Minimally Met
• List of all care and service providers
Not Met
in the MCP’s network (may be the
same as the provider directory) (AM) Not Applicable
Reviewer Notes:
• Organization strategic plans (AM)
• Administrative policies and
procedures (AM)
• Medicaid and CHIP and other
enrollee survey results (AM)
• Utilization management policies and
procedures (UM)
• Service authorization policies and
procedures (UM)
• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)
• Provider/Contractor oversight and
evaluation policies and procedures,
audit tools (PS)
• Medicaid and CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid and CHIP Enrollee
Handbooks (ES)
• Medicaid and CHIP provider directory
(ES)
• Medicaid and CHIP Enrollee
Orientation Curriculum (ES)
• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Assurances of
adequate capacity
and services
Medicaid: 42 CFR
438.207: Assurances
of adequate capacity
and services
CHIP: 42 CFR
457.1230(b):
Assurances of
adequate capacity
and services

• Medicaid and CHIP agency
documentation and submission timing
standards to assure that the MCP has
an appropriate range of preventive,
primary care, specialty, and LTSS
services that are adequate for the
anticipated number of enrollees in the
MCP’s service area.
• Medicaid and CHIP agency
documentation and submission timing
standards to assure that the MCP
maintains a network of providers that
is sufficient in number, mix, and
geographic distribution to meet the

144 | PROTOCOL THREE

• MCP 42 CFR 438.207(b) compliance
documentation

Medicaid:

• MCP 42 CFR 438.207(c) compliance
documentation

Substantially Met

• MCP 42 CFR 457.1230(b)

compliance documentation

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

needs of the anticipated number of
enrollees in the service area.

(See Worksheet
3.2 for
approaches to
compliance
scoring)
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Coordination and
The state’s requirements regarding the
continuity of care for obligation to and methods by which an
all enrollees
MCP must:
Medicaid: 42 CFR
• a) Ensure enrollees have an ongoing
source of care appropriate to their
438.208: Coordination
needs and a person or entity formally
and continuity of care
designated as primarily responsible for
CHIP: 42 CFR
coordinating the services accessed by
457.1230(c):
the enrollee. The enrollee must be
Coordination and
provided information on how to contact
continuity of care
their designated person or entity
• b) Coordinate the services
the MCP furnishes to enrollees
(between settings, between MCPs,
between MCP and FFS, and with
services provided by community and
social supports)

• Practice guidelines adopted by the
MCP (AM)

Medicaid:

• Provider/Contractor Services policies
and procedures manuals (PS)

Substantially Met

• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)

• e) Ensure that
each provider furnishing services
to enrollees maintains and shares, as
appropriate, an enrollee health record
in accordance with professional
standards

Partially Met
Minimally Met
Not Met

• Medicaid and CHIP enrollee services
policies and procedures (ES)

Not Applicable

• Medicaid and CHIP enrollment and
disenrollment policies and
procedures (ES)

CHIP:

Reviewer Notes:
Fully Met

• Medicaid and CHIP Enrollee
Handbooks (ES)

Substantially Met

• Care coordination policies and
procedures, and enrollee records
(ES)

Minimally Met

• c) Make a best effort to conduct an
initial screening of each enrollee's
• Sample of Medicaid and CHIP
needs, within 90 days of the effective
enrollee records (ES)
date of enrollment for all new enrollees
•
Medicaid and CHIP enrollment and
• d) Share with the state or other
disenrollment policies and
MCPs serving the enrollee the results
procedures (ES)
of any identification and assessment
of that enrollee's needs to prevent
duplication of those activities

Fully Met

Partially Met
Not Met
Not Applicable
Reviewer Notes:

• A copy of the state-MCP contract
provisions, which specify the
methods by which the MCP assures
the state Medicaid and CHIP agency
that it does not request disenrollment
for reasons other than those
permitted under the contract.

• f) Ensure that in the process of
coordinating care, each enrollee's
privacy is protected in accordance with
applicable privacy requirements
Additional
coordination and
continuity of care
requirements: LTSS

• Methods used by the Medicaid and
CHIP agency to identify to the MCP
enrollees who need LTSS.

• Whether the MCP is required to meet
identification, assessment, and
Medicaid: 42 CFR
treatment planning requirements for
438.208: Coordination
dually-enrolled beneficiaries.
and continuity of care

• Practice guidelines adopted by the
MCP (AM)

Medicaid:

• Provider/Contractor Services policies
and procedures manuals (PS)

Substantially Met

• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)

Fully Met
Partially Met
Minimally Met
Not Met

PROTOCOL THREE | 145

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)
CHIP: 42 CFR
457.1230(c):
Coordination and
continuity of care

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance
• Any Medicaid and CHIP agency LTSS
assessment mechanisms
requirements, including the
requirement to use appropriate
providers or individuals meeting the
Medicaid and CHIP agency’s LTSS
service coordination requirements.
• The state’s quality assurance and
utilization review standards.

Applicable MCP documents

(See Worksheet
3.2 for
approaches to
compliance
scoring)

• Enrollee services policies and
procedures (ES)

Not Applicable

• Enrollee Handbooks (ES)

CHIP:

• Care coordination policies and
procedures, and enrollee records
(ES)
• Sample of enrollee records (ES)

Reviewer Notes:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Additional
• Methods used by the Medicaid and
CHIP agency to identify to the MCP
coordination and
individuals with special health care
continuity of care
needs (SHCNs).
requirements: SHCN
• Whether the MCP is required to
Medicaid: 42 CFR
implement mechanisms for identifying,
438.208: Coordination
assessing, and producing a treatment
and continuity of care
plan for persons with SHCNs using the
CHIP: 42 CFR
state’s definition of SHCNs.
457.1230(c):
• Whether the MCP is required to meet
Coordination and
identification, assessment, and
continuity of care
treatment planning requirements for
dually-enrolled beneficiaries.
• Any Medicaid and CHIP agency
SHCN assessment mechanisms
requirements, including the
requirement to use appropriate
providers or individuals meeting the
Medicaid and CHIP agency’s LTSS
service coordination requirements.

• Practice guidelines adopted by the
MCP (AM)

Medicaid:

• Provider/Contractor Services policies
and procedures manuals (PS)

Substantially Met

• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)

Fully Met
Partially Met
Minimally Met
Not Met

• Enrollee services policies and
procedures (ES)

Not Applicable

• Enrollee Handbooks (ES)

CHIP:

• Care coordination policies and
procedures, and enrollee records
(ES)
• Sample of enrollee records (ES)

Reviewer Notes:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Whether the Medicaid and CHIP
agency requires the MCP to produce a
treatment or service plan
for enrollees with SHCN that are
determined through assessment to
need a course of treatment or regular
care monitoring.
• The state’s quality assurance and
utilization review standards.
Disenrollment
Medicaid: 42 CFR
438.56:
Disenrollment:
Requirements and
limitations
CHIP: 42 CFR
457.1212:
Disenrollment

• Obtain from the Medicaid and CHIP
agency Information on:
• Reasons for which the MCP may
request the disenrollment of an
enrollee.
• Methods by which the MCP assures
the Medicaid and CHIP agency that it
does not request disenrollment for
reasons other than those permitted
under the contract.

146 | PROTOCOL THREE

• Medicaid and CHIP enrollment and
disenrollment policies and
procedures (ES)

Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

• Whether the state chooses to limit
disenrollment.

CHIP:

• Medicaid and CHIP agency enrollee
disenrollment request policies.

Substantially Met

Fully Met
Partially Met

• Whether the Medicaid and CHIP
agency allows the MCP to process
enrollee requests for disenrollment.

Minimally Met
Not Met

• Whether the Medicaid and CHIP
agency requires enrollees to seek
redress through the MCP’s grievance
system before the Medicaid and CHIP
agency makes a disenrollment
determination on the enrollee’s
request.
Coverage and
authorization of
services
Medicaid: 42 CFR
438.210(a–e)*:
Coverage and
authorization of
services, including

(See Worksheet
3.2 for
approaches to
compliance
scoring)

• Obtain from the state any amount,
duration, and/or scope of service
requirements that are greater than
those set forth in 42 CFR 440.230 or,
for enrollees under the age of 21, as
set forth in 42 CFR Part 441, Subpart
B.

• Obtain from the state any statutory,
regulatory and policy definitions of
“medical necessity”, as well as any
42 CFR 440.230
quantitative and non-quantitative
Sufficiency of amount,
treatment limitation limits set forth in
duration, and scope;
those sources.
42 CFR Part 441,
•
Obtain from the state Medicaid and
Subpart B: Early and
CHIP agency the state-established
Periodic Screening,
standards for MCP processing of
Diagnosis, and
standard authorization decisions.
Treatment (EPSDT) of
•
Any Medicaid and CHIP agency drug
Individuals Under Age
authorization requirements, including
21;* and
whether the Medicaid and CHIP
42 CFR 438.114,
agency requires approval of outpatient
drugs before its dispensing under
Emergency and postSection 1927(d)(5)(A) of the Act.
stabilization services

Not Applicable
Reviewer Notes:

• Provider contracts (PS)

Medicaid:

• Contracts or written agreements with
organizational subcontractors (AM)

Fully Met

• Completed evaluations of entities
conducted before delegation is
granted (AM)
• Medicaid and CHIP and other
enrollee grievance and appeals data
(AM)
• Utilization management policies and
procedures (UM)

Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:

• Coverage rules and payment policies
(UM)

Fully Met

• Data on claims denials (UM)

Partially Met

• Service authorization policies and
procedures (standard, expedited and
extensions) (UM)
• Policies and procedures for notifying
providers and enrollees of denials of
service (UM)

Substantially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

CHIP: 42 CFR
457.1230(d):
Coverage and
authorization of
services
42 CFR 457.1228:
Emergency and poststabilization services
*Note: 42 CFR
438.210(a)(5),
438.210(b)(2)(iii),
440.230 and
441 Subpart B do not
apply to CHIP

PROTOCOL THREE | 147

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)
Information
requirements for all
enrollees
Medicaid: 42 CFR
438.100(b)(2)(i)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance
•

•

Enrollee right to
receive information in
accordance with 42
CFR 438.10:
Information
requirements
CHIP: 42 C.F.R
457.1220: Enrollee
rights
42 C.F.R 457.1207:
Information
requirements

Applicable MCP documents

• Medicaid and CHIP and other
Whether the Medicaid and CHIP
enrollee survey results (AM)
agency, enrollment broker, or MCP
must provide all required information • Provider contracts (PS)
to enrollees.
• Enrollee services policies and
procedures (ES)
Medicaid and CHIP agency
developed definitions for managed
• Statement of enrollee rights (ES)
care terminology, including appeal,
• Enrollee marketing materials
co-payment, durable medical
• Medicaid and CHIP marketing plans,
equipment, emergency medical
policies and procedures (ES)
condition, emergency medical
• Medicaid and CHIP enrollment and
transportation, emergency room
disenrollment policies and
care, emergency services, excluded
procedures (ES)
services, grievance, habilitation
services and devices, health
• Enrollee Handbooks (ES)
insurance, home health care,
• Enrollee grievance and appeals
hospice services, hospitalization,
policies and procedures (ES)
hospital outpatient care, medically
• Staff Handbooks (SP)
necessary, network, non• Staff Orientation and Training
participating provider, physician
Curriculum (SP)
services, plan, preauthorization,
• MCP provider directory (ES)
participating provider, premium,
prescription drug coverage,
• MCP Formulary (ES)
prescription drugs, primary care
• MCP website (ES)
physician, primary care provider,
provider, rehabilitation services and
devices, skilled nursing care,
specialist, and urgent care.

•

Medicaid and CHIP agency
developed model enrollee
handbooks and enrollee notices.

•

The language(s) that the Medicaid
and CHIP agency determines are
prevalent in the MCP’s geographic
service area, and all non-English
languages that the Medicaid and
CHIP identifies.

•

Policies relevant to written material
language and format, for example,
policies relevant to inclusion of
taglines.

•

Any interpretation services that the
Medicaid and CHIP agency makes
available to enrollees.

•

How the Medicaid and CHIP agency
defines ‘reasonable time’ for
purposes of providing the enrollee
handbook to enrollees.

•

Medicaid and CHIP agency
developed or approved language

148 | PROTOCOL THREE

(See Worksheet
3.2 for
approaches to
compliance
scoring)
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

(See Worksheet
3.2 for
approaches to
compliance
scoring)

describing grievance, appeal, and
fair hearing procedures and
timeframes, for inclusion in the
enrollee handbook.
•

Medicaid and CHIP agency policy
on whether enrollee are required to
pay costs for services while an
appeal or state fair hear is pending –
and the final decision is adverse to
the enrollee – for purposes of the
enrollee handbook.

•

Any content required by the state for
the enrollee handbook that is not
covered in 42 CFR 438.10(g).

•

Information on how the state has
defined a “significant change” in the
information MCPs are required to
give enrollees pursuant to 42 CFR
438.10(g).

•

Any applicable Medicaid and CHIP
laws on enrollee rights.

Enrollee right to
• Information on whether or not the
MCP has documented to the state any
receive information
moral or religious objection to
on available
providing, reimbursing for, or providing
treatment options
coverage of, a counseling or referral
Medicaid: 42 CFR
service for a particular Medicaid and
438.100(b)(2)(iii)
CHIP service or services.
Enrollee right to
receive information on
available treatment
options and
alternatives . . .
including
requirements of 42
CFR 38.102:
Provider-enrollee
communications

• Medicaid and CHIP and other
enrollee survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met

Partially Met

CHIP: 42 CFR
457.1222: Providerenrollee
communication

• Medicaid and CHIP Enrollee
Handbooks (ES)
• Medicaid and CHIP Enrollee
Orientation Curriculum (ES)

Not Met

• Medicaid and CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid and CHIP enrollee
marketing materials (ES)
• Medicaid and CHIP marketing plans,
policies and procedures (ES)
• Medicaid and CHIP enrollment and
disenrollment policies and
procedures (ES)

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Minimally Met
Not Applicable
Reviewer Notes:

• Staff Handbooks (SP)
• Staff Orientation and Training
Curriculum (SP)
Enrollee right to
participate in
decisions regarding
his or her care and

• A written description of any state
law(s) concerning advance directives.
The written description may include
information from state statutes on

• Medicaid and CHIP and other
enrollee survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met

Fully Met

PROTOCOL THREE | 149

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)
be free from any
form of restraint
Medicaid: 42 CFR
438.100(b)(2)(iv) and
(v): Enrollee right to:
- participate in
decisions regarding
his or her care,
including the right to
refuse treatment;
- Be free from any
form of restraint . . .
as specified in other
Federal regulations
And related:
42 CFR 438.3(j):
Advance directives

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

advance directives, regulations that
• Medicaid and CHIP enrollee services
implement the statutory provisions,
policies and procedures (ES)
opinions rendered by state courts and
• Statement of enrollee rights (ES)
other states administrative directives.
• Medicaid and CHIP enrollee
[Note to reviewers: Each state
marketing materials (ES)
Medicaid and CHIP agency is required
under Federal regulations at 42 CFR
431.20 to develop such a description
of state laws and to distribute it to all
MCPs. Revisions to this description as
a result of changes in State law are to
be sent to MCPs no later than 60 days
from the effective date of the change
in state law.]

(See Worksheet
3.2 for
approaches to
compliance
scoring)
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met

• Information on whether or not the
MCP has documented to the state any
moral or religious objection to fulfilling
the regulatory provisions pertaining to
advance directives

Not Applicable
Reviewer Notes:

CHIP: 42 CFR
457.1220: Enrollee
rights
Compliance with
other Federal and
state laws
Medicaid: 42 CFR
438.100(d):
Compliance with other
federal and state laws

• Obtain from the state Medicaid and
CHIP agency the identification of all
State laws that pertain to enrollee
rights and with which the state
Medicaid and CHIP Agency requires
its MCPs to comply.

• Medicaid and CHIP and other
enrollee survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met

• Medicaid and CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid and CHIP enrollee
marketing materials (ES)

CHIP: 42 CFR
457.1220: Enrollee
rights

• Medicaid and CHIP marketing plans,
policies and procedures (ES)
• Medicaid and CHIP enrollment and
disenrollment policies and
procedures (ES)

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met

• Medicaid and CHIP Enrollee
Handbooks (ES)

Partially Met

• Medicaid and CHIP Enrollee
Orientation Curriculum (ES)

Not Met

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Minimally Met
Not Applicable
Reviewer Notes:

• Staff Handbooks (SP)
• Staff Orientation and Training
Curriculum (SP)
Provider Selection
Medicaid: 42 CFR
438.214: Provider
selection

• Obtain from the state information on
any credentialing, re-credentialing, or
other provider selection and retention
requirements established by the state
that address acute, primary,

150 | PROTOCOL THREE

Medicaid:
• Service planning documents and
provider network planning documents Fully Met
(e.g., geographic assessments,
Substantially Met
provider network assessments,

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)
CHIP: 42 CFR
457.1233(a): Provider
selection

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance
behavioral, substance use disorder,
and LTSS providers, as appropriate.

Applicable MCP documents

(See Worksheet
3.2 for
approaches to
compliance
scoring)

enrollee demographic studies,
Partially Met
population needs assessments) (AM)
Minimally Met
• Contracts or written agreements with
Not Met
organizational subcontractors (AM)
Not Applicable
• Procedures and methodology for
Reviewer Notes:
oversight, monitoring, and review of
delegated activities (AM)
CHIP:
• Contracts or written agreements with
organizational subcontractors (AM)

Fully Met

• Completed evaluations of entities
conducted before delegation is
granted (AM)

Partially Met

• Provider/Contractor files, 15-20
individual health care professional
files, and 15-20 institutional provider
files (PS)
• Credentialing committee or other
provider review mechanism meeting
minutes (PS)

Substantially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Sample of files of practitioners who
have not been appointed or
reappointed (PS)
Sub-contractual
relationships and
delegation
Medicaid: 42 CFR
438.230:
Subcontractual
relationships and
delegation

• Obtain from the state the “periodic
schedule” established by the State
according to which the MCP is to
monitor and formally review on an
ongoing basis all subcontractors’
performance of any delegated
activities.

• Procedures and methodology for
oversight, monitoring, and review of
delegated activities (AM)
• Contracts or written agreements with
organizational subcontractors (AM)
• Completed evaluations of entities
conducted before delegation is
granted (AM)
• Ongoing evaluations of entities
performing delegated activities

CHIP: 42 CFR
457.1233(b):
Subcontractual
relationships and
delegation

Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Practice Guidelines
Medicaid: 42 CFR
438.236: Practice
guidelines
CHIP: 42 CFR
457.1233(c): Practice
guidelines

• Information on any state statutory,
regulatory, or policy requirements
concerning MCP practice guidelines.

• Provider contracts (PS)

Medicaid:

• Contracts or written agreements with
organizational subcontractors (AM)

Fully Met

• Practice guidelines (AM)
• Provider/Contractor Services policies
and procedures manuals (PS)

Substantially Met
Partially Met
Minimally Met
Not Met

PROTOCOL THREE | 151

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents
• Medicaid and CHIP enrollee services
policies and procedures (ES)

(See Worksheet
3.2 for
approaches to
compliance
scoring)
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Health information
systems
Medicaid: 42 CFR
438.242
CHIP: 42 CFR
457.1233(d):

• Information on whether or not the state
has required the MCP to undergo, or
has otherwise received, a recent
assessment of the MCP’s health
information system. If the state has
required or received such an
assessment, obtain a copy of the
information system assessment from
the state or the MCP. Also obtain
contact information about the person
or entity that conducted the
assessment and to whom follow-up
questions may be addressed.

• QAPI project descriptions, including
data sources and data audit results
(AM)
• Medicaid and CHIP and other
enrollee grievance and appeals data
(AM)
• Analytic reports of service utilization
(UM)
• Information systems capability
assessment reports (IS)

• Policies and procedures for auditing
data or descriptions of other
• State specifications for data on
mechanisms used to check the
enrollee and provider characteristics
accuracy and completeness of data
that must be collected by the MCP.
(internally generated and externally
generated data) information system
• Information on whether or not the state
has conducted a recent review and
• Completed audits of data or other
validation of the MCP’s encounter
evidence of data monitoring for
data, or required the MCP to undergo,
accuracy and completeness both for
or has otherwise received, a recent
MCP data and information system
validation of the MCP’s encounter
• Provider/Contractor Services policies
data. If the state has required or
and procedures manuals (PS)
received such a validation review,
•
Provider contracts (PS)
obtain a copy of the review from the
state or the MCP. Also obtain contact
information about the person or entity
that conducted the validation and to
whom follow-up questions may be
addressed.

• State specifications for how MCPs are
to (1) collect data elements necessary
to enable the mechanized claims
processing retrieval systems to
provide for electronic transmission of
claims data in the format consistent
with the Transformed Medicaid
Statistical Information System (TMSIS); (2) collect and transmit data on
enrollee and provider characteristics
specified by the state, on all services
furnished to enrollees through an

152 | PROTOCOL THREE

MCP Standards, Including Enrollee Rights and Protections
Reviewer
determination

Federal regulation
source(s)

Medicaid and CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

(See Worksheet
3.2 for
approaches to
compliance
scoring)

encounter data system; and (3)
Ensure that data received from
providers is accurate and complete.
• Specifications for submitting encounter
data to the Medicaid and CHIP agency
in standardized ASC X12N 837 and
NCPDP formats, and the ASC X12N
835 format.
• Make all collected data available to the
state and upon request to CMS.
• The state’s procedures and quality
assurance protocols to ensure that
enrollee encounter data submitted by
the MCP is a complete and accurate
representation of the services
provided to its enrollees.

Quality Assessment and Performance Improvement Program
Federal
regulation
source(s)
Quality
Assessment and
Performance
Improvement:
General rules
Medicaid: 42 CFR
438.330(a): General
rules
CHIP: 42 CFR
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation
information needed to
determine MCP compliance
• In the event that CMS specifies
national performance measures
or PIP topics, whether or not the
state has requested an exemption
from the national performance
measures or PIPs.

Applicable MCP documents
• MCP QAPI implementation
documentation (AM)

Reviewer determination
(See Worksheet 3.2 for
approaches to
compliance scoring)
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

PROTOCOL THREE | 153

Quality Assessment and Performance Improvement Program
Federal
regulation
source(s)
Basic elements of
quality
assessment and
performance
improvement
program
Medicaid: 42 CFR
438.330(b): Basic
elements of quality
assessment and
performance
improvement
programs
CHIP: 42 CFR
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation
information needed to
determine MCP compliance
• The state’s specifications for
performance improvement
projects (PIPs) required per
paragraph (d) of this section.
• The state’s specifications for how
the MCP should identify, measure
and report performance measures
required per paragraph (c) of this
section.
• The state’s requirements for
detection by the MCP of over- and
under-utilization.
• The state’s requirements for
assessment by the MCP of the
quality and appropriateness of
care furnished to enrollees with
special health care needs, as
defined in the state’s quality
strategy under 438.340 (as crossreferenced for CHIP in
457.1240(e)).
• The state’s requirements for
assessment by the MCP of the
quality and appropriateness of
care furnished using LTSS, if
applicable, including assessment
of care between care settings and
a comparison of services and
supports received with those set
forth in the enrollee’s
treatment/service plan.
• The state’s requirements for the
MCP’s participation in efforts by
the State to prevent, detect,
report, investigate and remediate
critical incidents, that occur within
the delivery of LTSS as well as to
track and trend results in order to
make systems improvements, if
applicable

154 | PROTOCOL THREE

Applicable MCP documents

Reviewer determination
(See Worksheet 3.2 for
approaches to
compliance scoring)

• Policies and procedures related to
QAPI project metrics (AM)

Medicaid:

• QAPI project quality indicators, the
selection or development criteria, and
processes for selection or development
(AM)

Substantially Met

• Performance standards and quality
indicators established by the MCP
(AM)
• Performance measure reports and data
provided to the state (AM)
• Utilization management policies and
procedures (UM)
• Medicaid and CHIP and other enrollee
LTSS tracking reports (AM)
• Policies and procedures related to data
collection and data quality checks for
QAPI projects (AM)
• Policies and procedures for
assessment of LTSS services between
care settings and comparison of
services and supports received with
those set forth in the enrollee's
treatment/service plan (AM)
• Policies and procedures for assisting
the state in the prevention, detection
and remediation of critical incidents
that occur within the delivery of LTSS.

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Quality Assessment and Performance Improvement Program
Federal
regulation
source(s)
Performance
measurement
Medicaid: 42 CFR
438.330(c):
Performance
measurement
CHIP: 42 CFR
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation
information needed to
determine MCP compliance
• Information on the standard
performance measures identified
by the state.

Applicable MCP documents
• Performance measure reports and data
provided to the state (AM)

Reviewer determination
(See Worksheet 3.2 for
approaches to
compliance scoring)
Medicaid:
Fully Met
Substantially Met

• For an MCP providing long-term
services and supports, the
standard performance measures
relating to quality of life,
rebalancing, and community
integration activities
for individuals receiving long-term
services and supports.

Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Information on whether the MCP
calculates the performance
measure and reports to the state
or whether the MCP provides data
to the state, which then calculates
the PM.

CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Performance
improvement
projects

• Information on any PIP
requirements specified by the
state.

Medicaid: 42 CFR
438.330(d) and

• Information on how often the state
requests that each MCP report
the status and results of each
project conducted per paragraph
(d)(1) of this section.

CHIP: 42 CFR
457.1240(b)

• Information on if the state permits
an MCP exclusively serving dual
eligibles to substitute an MA
Organization quality improvement
project conducted undeR
422.152(d) of this chapter for one
or more of the performance
improvement projects otherwise
required under this section.

• Reports and status documentation of
MCP internal QAPI evaluations (AM)

PROTOCOL THREE | 155

Quality Assessment and Performance Improvement Program
Federal
regulation
source(s)
QAPI evaluations
review
Medicaid: 42 CFR
438.330(e)(2):
Program and review
by the state
CHIP: 42 CFR
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation
information needed to
determine MCP compliance
• Information on whether the state
requires its MCPs to develop a
process to evaluate the impact
and effectiveness of its
own quality assessment and
performance improvement
program. If so, information on the
frequency with which that
evaluation must be conducted,
and on the state’s requirements
for how MCPs conduct that
process.

Applicable MCP documents
• Reports and status documentation of
MCP internal QAPI evaluations (AM)

156 | PROTOCOL THREE

Grievance System
Federal
Regulation
Source(s)

State Policy/Regulation
Information Needed to Determine
MCP Compliance

Grievance Systems

• Obtain information on:

Medicaid: 42 CFR
438.228: Grievance
and appeal systems

• Whether or not the Medicaid and
• Enrollee grievance and appeal
CHIP agency delegates
tracking reports (ES)
responsibility to the MCP for
providing each enrollee (who has
received an adverse decision with
respect to a request for a covered
service) notice that he or she has the
right to a state fair hearing or review
to reconsider their request for the
covered service.

Applicable MCP Documents
• Enrollee grievance and appeals
policies and procedures (ES)

Reviewer Determination
(See Worksheet 3.2 for
approaches to
compliance scoring)
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
Medicaid:

General
requirements

• Information on:

Medicaid: 42 CFR
438.402: General
requirements

Substantially Met
• Medicaid and CHIP and other enrollee
Partially Met
grievance and appeals data (AM)
• Whether providers, or authorized
Minimally Met
representatives, can act on behalf of • Analytic reports of service utilization
(UM)
the enrollee to request an appeal, file
Not Met
a grievance, or request a state fair
• Information systems capability
Not Applicable
hearing or review request.
assessment reports (information

CHIP: 42 CFR
457.1260: Grievance
system

• Whether enrollees are required or
permitted to file a grievance with
either the state or the MCP, or both.

• Whether state offers external
medical review.

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Fully Met

systems)

• Policies and procedures for auditing
data or descriptions of other
mechanisms used to check the
accuracy and completeness of both
internally generated and externally
generated data (Information systems)
• Completed audits of data or other
evidence of data monitoring for
accuracy and completeness both for
MCP data and contractor (delegate)
data (information systems)
• Provider/Contractor Services policies
and procedures manuals (PS)
• Provider contracts (PS)

Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes::

PROTOCOL THREE | 157

Grievance System
Federal
Regulation
Source(s)

State Policy/Regulation
Information Needed to Determine
MCP Compliance

Timely and
Adequate Notice of
Adverse Benefit
Determination

• Information on the timeframes within
which it requires MCPs to make
standard (initial) coverage and
authorization decisions and provide
written notice to requesting
enrollees. These timeframes will be
the required period within which
MCPs must provide Medicaid and
CHIP enrollees written notice of any
intent to deny or limit a service (for
which previous authorization has not
been given by the MCP) and the
enrollee’s right to file an MCP
appeal.

Medicaid: 42 CFR
438.404: Timely and
adequate notice of
adverse benefit
determination
CHIP: 42 CFR
457.1260: Grievance
system

Applicable MCP Documents

Reviewer Determination
(See Worksheet 3.2 for
approaches to
compliance scoring)

• Data on claims denials (UM)

Medicaid:

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Fully Met

• MCP adverse benefit determinations
(ES)
• Timing data on adverse benefit
determination mailings (ES)

Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Handling of
Grievances and
Appeals
Medicaid: 42 CFR
438.406: Handling of
grievances and
appeals

• Information on any state
requirements concerning handling of
grievances and appeals that differ
from those required under 438.406.
• *Note: See the ‘Disenrollment’
section in Worksheet 3.2 above for
grievances during disenrollment.

CHIP: 42 CFR
457.1260: Grievance
system

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

Medicaid:
Fully Met

Substantially Met
• Medicaid and CHIP and other enrollee
Partially Met
grievance and appeals data (AM)
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

158 | PROTOCOL THREE

Grievance System
Federal
Regulation
Source(s)

State Policy/Regulation
Information Needed to Determine
MCP Compliance

Resolution and
notification:
Grievances and
appeals

• Information on:

Medicaid: 42 CFR
438.408: Resolution
and notification,
Grievances and
appeals
CHIP: 42 CFR
457.1260: Grievance
system

Applicable MCP Documents
• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)

• The state-established standard time
frames during which the state
• Medicaid and CHIP enrollee
requires MCPs to (1) dispose of a
grievance and appeal tracking reports
grievance and notify the affected
(ES)
parties of the result, and (2) resolve
appeals and notify affected parties of • MCP appeal resolution notices (ES)
the decision.
• The methods prescribed by the state
that the MCP must follow to notify an
enrollee of the disposition of a
grievance.

Reviewer Determination
(See Worksheet 3.2 for
approaches to
compliance scoring)
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Information on whether providers, or
authorized representatives, can act
on behalf of the enrollee to request
an appeal, file a grievance, or
request a state fair hearing request.

CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Expedited resolution
of appeals
Medicaid: 42 CFR
438.410: Expedited
resolution of appeals
CHIP: 42 CFR
457.1260: Grievance
system

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)
• Medicaid and CHIP enrollee
grievance and appeal tracking reports
(ES)

PROTOCOL THREE | 159

Grievance System
Federal
Regulation
Source(s)

State Policy/Regulation
Information Needed to Determine
MCP Compliance

Information about
the grievance
system to providers
and subcontractors

• Information on:

Medicaid: 42 CFR
438.414: Information
about the grievance
and appeal system to
providers and
subcontractors
CHIP: 42 CFR
457.1260: Grievance
system

• Whether the state develops or
approves the MCP’s description of
its grievance system that the MCP is
required to provide to all Medicaid
and CHIP enrollees (per
438.10(g)(2)(xi). [Note that under
regulations at 42 CFR 438.10(g)(1)
the state must either develop a
description for use by the MCP or
approve a description developed by
the MCP.]

Applicable MCP Documents
• Contracts or written agreements with
organizational subcontractors (AM)

Reviewer Determination
(See Worksheet 3.2 for
approaches to
compliance scoring)
Medicaid:
Fully Met

• Completed evaluations of entities
conducted before delegation is
granted (AM)

Partially Met

• Provider contracts (PS)

Minimally Met

• Provider/Contractor procedure
manuals (PS)

Not Met

Substantially Met

Not Applicable
Reviewer Notes:

• If the states approves, rather than
develops, the description of the
MCP’s grievance system,
information on whether or not the
state has already approved the
MCP’s description.

CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Recordkeeping
requirements
Medicaid: 42 CFR
438.416:
Recordkeeping
requirements

• Information on any audits or other
reviews of MCP records of
grievances and appeals conducted
by the state.

CHIP: 42 CFR
457.1260: Grievance
system

• Medicaid and CHIP enrollee
grievance and appeals policies and
procedures (ES)
• Medicaid and CHIP enrollee
grievance and appeal tracking reports
(ES)
• Sample records of grievances and
appeals (ES)

160 | PROTOCOL THREE

Grievance System
Federal
Regulation
Source(s)

State Policy/Regulation
Information Needed to Determine
MCP Compliance

Continuation of
benefits while the
MCP appeal and the
state Fair Hearing
are pending

• Information on any state
requirements concerning
continuation of benefits pending
appeal and state fair hearing that
differ from those required under 42
CFR 420.

42 CFR 438.420:
Continuation of
benefits while the
MCO, PIHP, or PAHP
appeal and the state
fair hearing are
pending
(Note: This
requirement does not
apply to CHIP)
Effectuation of
reversed appeal
resolutions
Medicaid: 42 CFR
438.424: Effectuation
of reversed appeal
resolutions.
CHIP: 42 CFR
457.1260: Grievance
system

Applicable MCP Documents
• Medicaid enrollee grievance and
appeals policies and procedures (ES)

Reviewer Determination
(See Worksheet 3.2 for
approaches to
compliance scoring)
Medicaid-only:
Fully Met
Substantially Met
Partially Met
Minimally Met

• Information on any audits or other
reviews of MCP records of appeals
conducted by the state, to determine
MCP compliance with federal
continuation of benefits
requirements.

Not Met
Not Applicable
Reviewer Notes:

• Whether state permits managed care
plans to recover the cost of services.
See (d) reference to “state’s usual
policy.”
• Information on which entity- the state • Medicaid and CHIP enrollee
grievance and appeals policies and
or the MCP- is required to pay for
procedures (ES)
services when the state fair hearing
officer reversed a decision to deny
authorization of services, and the
enrollee received the disputed
services while the appeal was
pending.

MCP documents are identified using generic names, except in instances where the regulatory provisions refer to and
require a specific document be present and reviewed for content.
The subject matter of each applicable MCP document is indicated in parenthesis as follows:
AM = Administrative/
Managerial; PS = Provider/Contractor Services; UM = Utilization Management; ES = Enrollee Services; IS =
Information Systems; SP = Staff Planning, Education, Development and Evaluation.
The subject matter designation does not imply that the document cannot be used as a data source for addressing other
provision issues, or that it should be the sole source of data in evaluating compliance with the provisions noted.

PROTOCOL THREE | 161

Worksheet 3.2. Compliance Definitions
Instructions. This worksheet provides examples of compliance scoring and compliance definitions. Either method
can be used by a state to distinguish levels of MCP compliance.
1.

Compliance Scoring. One commonly used approach to analyzing compliance review findings is to assign a
numerical value to indicate the degree of compliance with a given regulatory provision. An EQRO can provide a
compliance score for each regulatory provision on Worksheet 3.1. Compliance Review, followed by details and
justification for the compliance determination. Before the review, the state is directed to define what constitutes
compliance and determine the rating or scoring system for what the EQRO will review. The state and EQRO can
adapt a compliance rating scale to best suit their needs. Some examples include:
 Two-point rating or scoring. Either the requirement is met or not met:
•

Met = 1

•

Not Met = 2

 Three-point rating or scoring. This scale provides credit when a requirement is partially met:
•

Fully Met = 1

•

Partially Met = 2

•

Not Met = 3

 Five-point rating or scoring. This scale allows for the scoring of all five levels of compliance:
•

Fully Met = 1

•

Substantially Met = 2

•

Partially Met = 3

•

Minimally Met = 4

•

Not Met = 5

One of the above rating or scoring scales may serve as the primary system, or alternative scales may be adapted to
certain regulatory provisions. In an extensive compliance review, the state may assert that the definition of
compliance for most regulatory provisions are appropriate for a 5-point rating scale, with two or three particular
provisions rated as “met” or “unmet.”
2. Compliance Definitions Options. The following definitions describe the extent of compliance with a given
regulatory provision:
 Full compliance:
•

All documentation listed under a regulatory provision, or component thereof, is present, and

•

MCP staff provide responses to the EQRO that are consistent with each other and with the documentation;
or

•

A state-defined percentage of all data sources—either documents or MCP staff—provide evidence of
compliance with regulatory provisions

162 | PROTOCOL THREE

 Substantial Compliance:
•

After review of the documentation and discussion with MCP staff, it is determined that the MCP has met
most of the requirements as stated above

 Partial Compliance:
•

All documentation listed under a regulatory provision, or component thereof, is present, but MCP staff are
unable to consistently articulate evidence of compliance; or

•

MCP staff can describe and verify the existence of compliant practices during the interview(s), but required
documentation is incomplete or inconsistent with practice; or

•

Any combination of “Met,” “Partially Met” and “Not Met” determinations for smaller components of a
regulatory provision would result in a “Partially Met” designation for the provision as a whole

 Minimal Compliance:
•

After review of the documentation and discussion with MCP staff, it is determined that although some
requirements have been met, the MCP has not met most of the requirements

 Non-compliance:
•

No documentation is present and MCP staff have little to no knowledge of processes or issues that comply
with regulatory provisions; or

•

No documentation is present and MCP staff have little to no knowledge of processes or issues that comply
with key components (as identified by the state) of a multi-component regulatory provision, regardless of
compliance determinations for remaining, non-key components of the regulatory provision

About Targeted Regulatory Components
If all applicable federal requirements are met, the state may focus on specific aspects or components of its regulatory
provisions to make performance improvement more manageable and targeted. If less than full compliance with a full
set of state regulations is defined by the state as acceptable, the state must identify to the EQRO and the MCP
specific regulatory provisions of the compliance review for which the MCP is accountable. This must take place
before the review begins. However, over the three-year compliance review cycle, the EQRO must review all
compliance requirements.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL THREE | 163

Worksheet 3.3. Sample Site Visit Agenda
Instructions. This worksheet provides a template to develop the site visit agenda. The agenda is intended to help
both the MCP and EQRO in planning staff interviews, gathering documentation, and determining logistics (such as
meeting space).
The EQRO should prepare the agenda and send it in advance to the individual representing the MCP in the
regulatory compliance review process. The MCP representative is responsible for identifying additional MCP
participants. The agenda should also provide the locations where the meetings and any additional document review
will occur. The EQRO should determine the number of days for the site visit based on the estimated duration and
number of meetings required to carry out the site visit. At the end of each day, the EQRO should lead a concluding
meeting to discuss outstanding information and answer questions. At the end of the final day, the EQRO should
provide concluding remarks and identify next steps in the review process.
Sample EQRO Site Visit Agenda (Add more days as needed)
•

Introductions between the EQRO reviewers and MCP participants

•

Site Visit Purpose: To clarify MCP compliance with federal Medicaid managed care regulations

•

Day 1 Activities: [Insert number of interviews and document review, if applicable]

•

Day 1 Final Meeting [Insert time]: Identification/discussion of outstanding issues
Regulatory
compliance issue

Time

Location

Participants

Comments/Documents

(EQRO)

(MCP)

(BOTH)

(EQRO)

(MCP)

(BOTH)

(EQRO)

Add more rows as
needed
Note:

The organization responsible for completing each part of the agenda is identified above in parenthesis.

•

Day 2 Activities: [Insert number of interviews and document review, if applicable]

•

Day 2 Final Meeting [Insert time]: Concluding issues and comments; description of next steps

•

Day 2 Exit Interviews [Insert time]: Opportunity to respond to initial compliance issues (if applicable) and clarify
reviewer understanding
Time

Regulatory
compliance issue

Location

Participants

Comments/Documents

(EQRO)

(MCP)

(BOTH)

(EQRO)

(MCP)

(BOTH)

(EQRO)

Add more rows as
needed
*The organization responsible for completing each part of the agenda is identified above in parenthesis.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

164 | PROTOCOL THREE

Worksheet 3.4. Compliance Interview Questions
Instructions. Worksheet 3.4 includes a list of questions the reviewer may ask MCP staff to help determine
compliance with state and federal requirements as summarized in Activity 3, Step 6. The purpose of the MCP
interviews is to collect data to supplement and verify what is learned through the preliminary document review and
onsite or virtual site visit document review. The questions are first organized by MCP staff roles, and then broken out
by regulatory provision.
Reviewers are encouraged to interview MCP staff in appropriate groups whenever possible in order to accomplish a
comprehensive review from more than one perspective, and to achieve efficient and productive interviews.
The MCP interviewee groups who are most often interviewed are included in this guide:
•

MCP leaders

•

MCP information systems staff

•

Quality assessment and performance improvement program staff

•

Provider/contractor services staff

•

Enrollee services staff

•

Utilization management staff

•

Medical directors

•

Case managers and care coordinators

•

MCP providers and contractors (as appropriate)

The EQRO should advise the MCP of the specific issues for which the MCP will be interviewed during the site visit.
The MCP representative for the compliance review process should select and report to the EQRO in writing the
membership of each of the interviewee groups that are capable of responding to the EQRO site visit interview topic
requests.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL THREE | 165

MCP Leaders Interview
The leadership interview is an opportunity to speak with the senior representatives of the MCP about their
understanding of MCP requirements. MCP leaders include:
•

Chief executive officer (CEO)

•

Chief operating officer (COO)

•

Chairman of the governing body, or a representative

•

Medical director (including psychiatric medical director, if applicable)

•

Chief elected or appointed officer of the MCP 's licensed independent practitioners

•

Chief information officer (CIO)

•

Compliance officer

•

Quality improvement committee chairperson

•

Quality improvement program director or coordinator, and

•

Human resources leader

As determined by the MCP representative, usually in consultation with the CEO, other senior staff of the MCP may
also be in attendance. However, attendance at this interview should be carefully limited in order to foster candor and
exchange of information.
I. MCP Standards and Enrollee Rights and Protections
Availability of services (42 CFR 438.206 and 457.1230(a))
1.

Please describe the MCP’s process for assessing whether its network of appropriate providers 59 is sufficient
to provide adequate access to each type of covered service and major specialty within each type of covered
service.
a.

What issues were considered in the assessment process?

How does the MCP determine the adequacy of its network to serve its Medicaid and CHIP enrollees?

What assumptions and methodologies are used to project the number, type (training, experience, and
specialization), and location of primary care providers and specialists necessary to serve Medicaid and
CHIP enrollees?

What assumptions and methodologies are used to project the number, type (training, experience, and
specialization), and location of LTSS providers necessary to serve Medicaid enrollees?

If the state has established access requirements for LTSS, how does the MCP evaluate its current network
in comparison to the requirements?

Per 42 CFR 438.2 and 457.10, provider means any individual or entity that is engaged in the delivery of services, or ordering
or referring for those services, and is legally authorized to do so by the state in which it delivers the services.

166 | PROTOCOL THREE

Are there any areas where the requirements are not met? If so, how is the MCP remedying these gaps?

Out-of-network providers (42 CFR 438.206(b)(3) through (5) and 457.1230(a))
1.

Approximately what proportion of Medicaid and CHIP enrollee provider encounters are made to out-ofnetwork providers? If this is a significant percent, what are the reasons for this?

What are the reimbursement methods for out-of-network providers? Which types of providers are paid using
each method?
[Probe: Do you receive claim, encounter data from out-of- network providers similar to the claim, or
encounter data that you receive from your network providers?]

How does your MCP ensure that any costs to the Medicaid and CHIP enrollee for out-of-network services is
no greater than the costs the enrollee would incur if they used a network provider for the same service?

Furnishing of services and timely access (42 CFR 438.206(c)(1) and 457.1230(a))
1.

Please describe how the MCP monitors compliance with its Medicaid and CHIP standards for timely access
to care and services.

How does the MCP ensure the 24 hours per day, 7 day per week availability of Medicaid and CHIP services
included in its contract with the state when medically necessary?

How does the MCP determine that the individual and institutional providers it contracts with have sufficient
capacity to make services available when medically appropriate 24 hour per day, 7 days per week to
Medicaid and CHIP enrollees?

How does the MCP ensure that its provider network’s hours of operation do not discriminate against
Medicaid and CHIP enrollees (i.e., are not different for Medicaid and CHIP enrollees than for commercial
enrollees)?

How is inappropriate use of emergency department visits addressed? What proportion of emergency
department visits are potentially avoidable?

What was the volume of denied claims for emergency and post-stabilization services in the most recent
year?

Access and cultural considerations (42 CFR 438.206(c)(2) and 457.1230(a))
1.

What have been the state Medicaid and CHIP agency’s efforts to promote the delivery of services in a
culturally appropriate manner to all enrollees, including those with diverse cultural and ethnic backgrounds?
a. How has your MCP participated in these efforts?
b. What documentation exists describing your efforts and the results of these efforts?

What efforts has the MCP made to promote services to enrollees with limited English proficiency and those
with low literacy?

How does the MCP maintain and make available information on all languages (including both spoken and
signed) used by providers, including those used by LTSS providers?

PROTOCOL THREE | 167

How are call center staff made aware of MCP beneficiaries’ needs so that verbal communication is easily
understood by the beneficiary? For example, volume or speed of speech.

Assurances of adequate capacity and services (42 CFR 438.207(b) – (c) and 457.1230(b))
1.

Please describe how your MCP demonstrates to the Medicaid and CHIP agency that it offers an appropriate
range of preventive, primary care, specialty services, and LTSS that is adequate for the anticipated number
of enrollees in your service area.

Please describe how your MCP demonstrates to the Medicaid and CHIP agency that it maintains a network
of providers that is sufficient in number, mix, and geographic distribution to meet the needs of the anticipated
number of enrollees in the service area.

Please indicate whether your MCP meets the timing standards set by the Medicaid and CHIP agency for this
documentation.

Coordination and continuity of care for all enrollees (42 CFR 438.208, 457.1230(c))
1.

Which provider types are authorized by the Medicaid or CHIP agency to serve as enrollee primary care
providers? (i.e., general practitioner, family physician, internal medicine physician, OB/GYN, pediatrician, or
other licensed practitioner as authorized by the state Medicaid program)
a.

Does your MCP permit each of the provider types authorized by the Medicaid or CHIP agency to
provide primary care services to serve as primary care providers?

What steps does the MCP take to promote Medicaid and CHIP enrollees’ ongoing relationship with a usual
source of primary care?

What processes are used to coordinate services for enrollees?
a.

Are there different types of care coordination mechanisms for different types of enrollees? If so, what
are these?

Are there different types of care coordination mechanisms for acute and primary services? If so, what
are these?

If your MCP establishes separate coordination of care for medical services, LTSS, and mental health and
substance abuse services, how does it ensure exchange of necessary information between care
coordinators? How does it ensure information exchange among providers?

How are staff trained in the processes and tools required to facilitate integrated medical, behavioral, care
planning, service planning, and authorization activities?

How does the MCP ensure coordination of its services with services enrollees may receive from other MCPs
or community programs and providers?

How are coordination and communication ensured when an enrollee changes MCPs or transitions
between FFS and managed care?

How are coordination and communication ensured when an enrollee is a member of more than one
MCP (e.g., duals and separate dental or behavioral health plans)?

Under what circumstances may Medicaid and CHIP enrollees have direct access to specialists?

168 | PROTOCOL THREE

How does your MCP manage access to any specialty care services currently not provided in-network?

Does your MCP require written treatment plans to be developed for enrollees? If yes, under what
circumstances are written treatment plans required?

Additional coordination and continuity of care questions: LTSS (42 CFR 438.208 and 457.1230(c))
1.

Does your state’s Medicaid and CHIP agency require your MCP to meet identification, assessment, and
treatment planning requirements for dually-enrolled beneficiaries who need LTSS?

Please describe how your MCP meets any LTSS assessment mechanism requirements established by your
Medicaid and CHIP agency.

What processes are used to coordinate services for enrollees who need LTSS?
a.

Are there different types of care coordination mechanisms for LTSS as compared to acute and primary
care services? If so, what are these?

If your MCP establishes separate coordination of care for LTSS, how does it ensure exchange of necessary
information between care coordinators? How does it ensure information exchange among providers?

How are staff trained in the processes and tools required to facilitate integrated medical, behavioral and
LTSS assessment, care planning, service planning, and authorization activities for enrollees who need
LTSS?

How are coordination and communication ensured when an LTSS enrollee is a member of more than one
MCP?

Additional coordination and continuity of care questions: SHCN (42 CFR 438.208 and 457.1230(c))
1.

How are “individuals with special health care needs” defined by the state Medicaid and CHIP agency?
a.
b.

Has your MCP developed any other operational definition or definitions of individuals with special health
care needs?
If yes, what is/are these and how were they developed? How do they differ from the state definition?

Does the state Medicaid or CHIP agency require your MCP to screen Medicaid and CHIP enrollees to
identify those with special health care needs?

How are individuals with special health care needs—including both individuals with special health care
needs identified by this MCP and those identified by the state Medicaid or CHIP agency or its agent—
identified and tracked within your MCP?

Does the state Medicaid agency require your MCP to assess and provide treatment plans for Medicaid and
CHIP enrollees with special health care needs? If yes, how are these activities conducted?

Disenrollment (42 CFR 438.56 and 457.1212)
1.

For what reasons may your MCP request the disenrollment of an enrollee?
a.

Has your MCP requested to disenroll an enrollee for any other reason?

Is your MCP allowed to process enrollee disenrollment requests? If so, for what reasons have enrollees
requested disenrollment?

PROTOCOL THREE | 169

Are enrollees required to seek redress through your MCP’s grievance system before a disenrollment request
is determined? If so, what timeframes are used by your MCP to process those grievances?

Enrollee right to information (42 CFR 438.100 and 457.1220; 42 CFR 438.10 and 457.1207)
1.

How does your MCP provide written notice of any change (that the state defines as “significant”) to the
information contained in the enrollee handbook, at least 30 days before the intended effective date of the
change?
a.

How does the state define “significant”?

Have you made any such “significant” changes in the last year? If yes, what were those changes?

How do you ensure that your staff and affiliated providers comply with federal and state laws that apply to
enrollee rights?

What information is routinely provided to Medicaid and CHIP enrollees?
a.

What is the process for disseminating information to new and existing enrollees?

How often is information distributed to existing enrollees?

In what format is this information presented?

Please describe or provide copies of the formats in which information is presented to enrollees.

In what languages or alternative formats are enrollee materials and information presented? If other
languages or alternative formats are used, how was it determined that materials were needed in different
languages or formats?

Does the MCP provide written materials in alternative formats for the visually impaired? If yes, how did the
MCP determine that materials were needed for the visually impaired?

Please describe the procedures for handling calls to the MCP from non-English speaking enrollees.
a.

What instruction or guidance is available for providers that may need interpretation assistance to
provide care and services to assigned enrollees?

To what extent is the MCP responsible for responding to requests for information for potential Medicaid and
CHIP enrollees?

How does the MCP inform enrollees (and potential enrollees, if applicable) about how to obtain oral
interpreter services if they have limited proficiency in English?

10. Are there any benefits that an enrollee is entitled to under the Medicaid and CHP program, including LTSS
benefits, but that are not made available through the MCP contract? If yes, what are those benefits? How
are enrollees made aware of the Medicaid and CHIP program benefits that are outside the scope of services
available through the MCP?
11. How does the MCP ascertain the primary language spoken by the individual Medicaid and CHIP enrollees?
12. Are enrollees provided with a listing of primary care providers? If yes, does this listing include providers’
non-English language capabilities?

170 | PROTOCOL THREE

13. Does the MCP give written notice of the termination of a contracted provider to enrollees who receive
primary care from, or are seen on a regular basis by, the terminated providers? If yes, how is this
accomplished? Have you had to make any such notifications in the last year?
14. How does the MCP ensure that information and instructional materials intended for enrollees and potential
enrollees are easily understood by those with a variety of cognitive and intellectual capabilities?
15. How does the MCP provide its enrollees information about provider appeal rights regarding coverage of a
service?
16. Does the MCP provide information to providers on where to refer enrollees who are having difficulty
understanding the materials that have been provided to them by the MCP?
17. What protocols does the MCP follow to develop materials that are readily understandable by enrollees?
18. Does the MCP require providers to have access to oral interpreter services?
19. Does the MCP provide providers with guidance or assistance in accessing interpreter services if necessary?
Enrollee right to respect, dignity, privacy (42 CFR 438.100 457.1220)
1.

How does the MCP ensure that its own facilities and those of its affiliated providers comply with enrollee
rights such as treatment with respect, dignity, and consideration for privacy and confidentiality of
information? Please provide an example?
a.

Are there any additional considerations made for providers of LTSS, or other specialized providers,
where services may be of a more intimate nature or occur in a more isolated setting? Please provide an
example?

What processes are in place to ensure that staff members observe the MCP’s policies and procedures on
privacy and confidentiality of enrollee information?

What does the MCP do to educate staff about policies on nondiscriminatory and culturally appropriate
behavior towards enrollees?
a.

How do you monitor staff compliance with these policies?

Enrollee right to receive information on available treatment options (42 CFR 438.100, 438.102; and 42 CFR
457.1220 and 457.1222)
1. How does the MCP ensure that providers share information on available treatment options and alternatives
with enrollees?

Does this include alternatives and options that are both within and outside the Medicaid or CHIP
contract scope of benefits?

How does the MCP ensure providers share information about HCBS as alternatives to institutional
care?

What steps does the MCP take to ensure that enrollees receive information on available treatment options
and alternatives in a manner appropriate to their condition and ability to understand?

PROTOCOL THREE | 171

Enrollee right to participate in decisions regarding his/her health care and advance directives (42 CFR
438.100 and 42 CFR 438.6; 42 CFR 457.1220)
1.

How does the MCP facilitate enrollee participation in care and treatment decisions? Please describe. Could
you provide an example?

Does the MCP have any limitations in implementing federal and state laws that apply to advance directives?
If so, what are these limitations?

Enrollee rights (42 CFR 438.10 and 457.1207) and Enrollee information (42 CFR 438.100 and 457.1220, and 42
CFR 438.206-210 and 457.1230(a–d))
1.

Does the MCP provide information to providers on where to refer enrollees who are having difficulty
understanding the materials that have been provided to them by the MCP?

What protocols does the MCP follow to develop materials that are readily understandable by enrollees?

Does the MCP require providers to have access to oral interpreter services?

Does the MCP provide providers with guidance or assistance is accessing interpreter services if necessary?

Compliance with other federal and state laws (42 CFR 438.100 and 457.1220, and 42 CFR 438.206-210 and
457.1230(a–d))
1.

What steps do MCP leaders take to ensure compliance with federal and state laws on enrollee rights?

Has the MCP ever been found non-compliant with any federal and state laws on enrollee rights? If yes, in
what area? What steps were taken to clear the violation?

If a provider/contractor/sub-contractor is found to be in violation of any federal and state laws on enrollee
rights, how does the MCP respond?

To what extent does the MCP orient new staff to federal and state laws on enrollee rights that must be
observed during day-to-day operations?
a.

How does the MCP remind staff of the importance of observing these laws during interactions with other
employees and with enrollees?

Please describe the steps taken by the MCP when staff report, or are involved in a violation of federal or
state laws on enrollee rights.

Coverage and authorization of services, including emergency and post-stabilization services (42 CFR
438.210 and 457.1230(d) and 42 CFR 438.114 and 457.1228)
1.

What percent of emergency department care utilized by your Medicaid and CHIP enrollees is for non-urgent
care?

Has your MCP investigated a potential relationship between inappropriate emergency department use and
enrollee access to routine and urgent care, or reviewed the most frequent diagnoses resulting in
inappropriate emergency department use?

What was the rate of denied claims for emergency and post-stabilization services in the most recent year?

172 | PROTOCOL THREE

What was the rate of appeals for denied claims for emergency and post-stabilization services in the most
recent year?
a.

Of these appeals, what was the rate in which claim denials were overturned?

What is the average wait time for MCP enrollees who see emergency services?

How many urgent care clinics with non-traditional hours are in the MCP’s network?

How does the MCP inform enrollees of emergency coverage?

Are emergency back-up plans created for all enrollees? If not, how is the need for an emergency back-up
plan determined? How is the emergency back-up plan shared with all appropriate parties?

Are certain LTSS providers or other specialized providers/provider types contracted specifically for afterhours/urgent/emergent need? If so, what types? How were these types determined?

10. How does the MCP ensure that it provides services in a sufficient amount, duration, and scope consistent
with contract requirements?
11. What are the MCP’s policies on service and drug limitations? What services or drugs does it limit?
a.

How does the MCP ensure that limited services can still reasonably achieve their purpose?

How does the MCP ensure that services supporting individuals with ongoing or chronic conditions, or
who require LTSS, are authorized in a manner that reflects the enrollee’s ongoing need for such
services and supports?

How does the MCP ensure that family planning services are provided in a manner that protects the
enrollee’s freedom to choose their preferred method?

12. How does the MCP ensure that it providers all medically necessary services specified by the contract?
13. What mechanisms does the MCP use to ensure consistent application of authorization decision review
criteria?
14. What mechanisms does the MCP use to notify providers and enrollees of adverse benefit determinations?
a.

What timeframes does the MCP use to process standard and expedited authorization decisions?

15. What notice methods does the MCP use for outpatient drug authorization decisions?
Provider Selection and Non-Discrimination (42 CFR 438.214 and 457.1233(a) and 42 CFR 438.12 and 457.1208)
1.

What is the basis or criteria used to determine individual provider participation in the MCP’s network?

What is the basis or criteria used to determine institutional or other non-individual practitioner (including
LTSS) participation in the MCP’s network?

What types of providers are subject to the MCP’s credentialing process?
a.

How are provider qualifications (including background check requirements) verified for provider types
not subject to the credentialing process?

PROTOCOL THREE | 173

Please describe the provider credentialing process used by the MCP.

What steps does the MCP take to ensure that it does not employ or contract with providers who have been
excluded from participation in federal health care programs?

What steps does the MCP take to ensure that providers who serve high-risk or costly populations are not
discriminated against in the selection process, and when considering reimbursement and indemnification?

What criteria is the basis for denial of provider participation in the MCP’s network?

Sub-Contractual Relationships and Delegation (42 CFR 438.230 and 42 CFR 457.1233(b))
1.

What services and activities are delegated to and performed by sub-contractors?

Please describe the MCP’s process for identifying and selecting contractors. How is it determined that a
contractor has the ability to provide the sub-contracted services?

Please describe how your MCP assesses the quality of sub-contracted services and sub-contractor
compliance with federal, state and contractual requirements.

Practice guidelines: adoption (42 CFR 438.236(b) and 457.1233(c))
1.

What organizational component of your MCP is responsible for the adoption of practice guidelines used by
your MCP?

How does your MCP establish priorities for adoption of practice guidelines?
a.

How does your MCP consider the enrolled Medicaid and CHIP population’s health needs in the
adoption of practice guidelines?

What guidelines has your MCP adopted?

By what institutional process were they adopted?

To what extent are your MCP’s guidelines “evidence-based”? By evidence-based, we mean systematically
developed statements to assist practitioner and patient decisions about appropriate health care for specific
clinical circumstances.

How does your MCP consider the enrolled Medicaid and CHIP population’s health needs in the adoption of
practice guidelines?

How are affiliated providers consulted as guidelines are adopted and re-evaluated?

What mechanism(s) does your MCP have for periodically evaluating and updating the guidelines it has
adopted?

Practice guidelines: dissemination and application (42 CFR 438.236(c) and 457.1233(c))
1.

How are practice guidelines disseminated to providers?

When and how are guidelines disseminated to enrollees and potential enrollees?

To what extent are the practice guidelines adopted by your MCP a component of your MCP’s Quality
Assessment and Performance Improvement (QAPI) program?

174 | PROTOCOL THREE

Is there a process in place to ensure communication between those responsible for the QAPI program and
the practice guidelines adoption process?

What steps are taken to ensure that decision-making in the areas of utilization management or coverage
determinations and other functional areas are consistent with the adopted practice guidelines?

Health information systems (42 CFR 438.242 and 457.1233(d))
1.

Describe the types of data collection systems that are in place to support the clinical and administrative
operations of your MCP. Specifically, what data is routinely collected to support utilization management,
grievance systems, and enrollment services?

What processes are in place to obtain data from all components of your network (e.g., health care facilities,
physician, laboratories, and LTSS, and other specialized providers)?

To what extent does your MCP require and receive data in standardized formats?

Are there any components of your network from which you do not receive standardized (or any)
information on services?

How are enrollee and provider data collected and integrated across all components of your MCP’s network?
a.

How is this used to produce comprehensive information on enrollee needs and utilization and to
otherwise support management?

II. Quality Assessment and Performance Improvement Program
Quality assessment and performance improvement program: general rules and basic elements (42 CFR
438.330 and 457.1240(b))
1.

Does the state require the MCP to address a specific topic or topics in your performance improvement
projects? If yes, what types of projects are required? For each PIP, at a minimum, include how significant
improvement was measured, how improvement will be/was sustained, and how beneficiary health outcomes
and satisfaction will be/was measured, and how the intervention will/has improved access and/or quality of
care?
a.

For duals-only MCPs, was a Medicare Advantage PIP substituted for a state-required PIP?

Has CMS specified any specific PIPs? If yes, what types of projects are required? For each PIP, at a
minimum, include how significant improvement was measured, how improvement will be/was sustained,
and how beneficiary health outcomes and satisfaction will be/was measured, and how the intervention
will/has improved access and/or quality of care?

Does the state require your MCP to collect and submit performance measures or to submit data to the state
for it to calculate performance measures? If yes, what performance measures are specified by the state and
who calculates each measure, the MCP or the state?
a.

If CMS specifies any performance measures, what performance measures are collected and submitted,
if any?

If the MCP provides LTSS, what LTSS performance measures are collected and submitted, including
but not limited to measures of quality of life, rebalancing institutional and community-based services and
community integration activities?

PROTOCOL THREE | 175

How does the MCP detect over- and under-utilization? Please provide examples of how your quality
assessment and improvement program has monitored to detect under- and over-utilization. What standards
and measures are used?

How does the MCP define enrollees with “special health care needs”? Does this definition match the state’s
definition of special health care needs? How are these enrollees identified/ tracked within your MCP?

How does the MCP assess the quality and appropriateness of care including LTSS, furnished to enrollees
with special health care needs? Please provide examples.

Does the state require the MCP to evaluate the impact and effectiveness of its quality assessment and
performance improvement program?

How does the MCP conduct its evaluation? What aspects of the program are included in the
evaluation?

How often does the MCP conduct its evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of the findings?

What is reported to the state, and how often?

For MCPs that provide LTSS services:
a.

How does the MCP assess quality and appropriateness of care in general, including but not limited to
between care settings and comparing treatment plans to service/supports received?

How does the MCP participate in the state’s efforts to prevent, detect, and remediate critical incidents?

What interventions are used or are anticipated to be used to improve LTSS quality? How will the
interventions be evaluated for effectiveness? How will improvement be sustained or increased?

Quality assessment and performance improvement program: program review by the state (42 CFR 438.330(e)
and 457.1240(b))
1.

How does the state review the impact and effectiveness of the MCP’s QAPI program, including outcomes
and trended results from the PIPs, reporting on performance measures, and the results of community
integration for beneficiaries receiving LTSS?
a.

What is the MCP’s role in the state’s evaluation?

What information, if any, does the MCP provide to the state?

What feedback, if any, does the MCP receive from the state? How does the MCP implement the
feedback?

III. Grievance System
Grievance system: denial of services (42 CFR 438.228)
1.

How does the MCP track requests for covered services that the MCP or its providers have denied?

176 | PROTOCOL THREE

What was the volume of denied claims for services in the most recent year?

How do you ensure that Medicaid enrollees who were denied services were notified of their right to a state
fair hearing?

Grievance system: general requirements (42 CFR 438.402 and 457.1260)
1.

Who in the MCP is responsible for the development and oversight of the appeals and grievance resolution
process and access to state fair hearings or review?

What have been the volume of appeals/grievances/requests for state fair hearings or reviews in the past
year and the most common areas of concern expressed by Medicaid and CHIP enrollees?
a.

How has the MCP addressed these concerns?

Describe the notice and appeals process for adverse actions on enrollee requests for services or payment.
Please describe the particular steps, including time frames.

Grievance systems: continuation of benefits (42 CFR 438.404(b)(6))
1.

Does the Medicaid enrollee's right to have benefits continue pending resolution of the appeal, the process to
request that benefits be continued, and the circumstances under which the enrollee may be required to pay
the costs of these services differ between medical and LTSS? Note that continuation of benefits
requirements do not apply to CHIP enrollees.
a.

If so, how?

Are there any special considerations required for continuation of LTSS pending resolution of an appeal?

Handling of grievances and appeals (42 CFR 438.406 and 457.1260)
1.

To what extent does your MCP provide Medicaid and CHIP enrollees with assistance in completing forms
and taking other procedural steps in the grievance and appeal process? How does the MCP provide
assistance?

How does your MCP treat oral requests by Medicaid and CHIP enrollees to appeal actions?

As part of an appeal, to what extent do enrollees and their representatives have an opportunity to:

Present evidence, and

Examine the enrollee’s case file, including medical records, and any other documents and records
considered during the appeals process.

What are the qualifications and credentials of individuals who make decisions on grievances and appeals?
a.

How does the MCP ensure that these individuals have not been involved in any previous level of review
or decision-making?

How does the MCP ensure that these individuals have the appropriate clinical expertise in treating the
enrollee’s condition or disease, if deciding any of the following:

PROTOCOL THREE | 177

An appeal of a denial that is based on lack of medical necessity

ii.

A grievance regarding denial of expedited resolution of an appeal

iii.

A grievance or appeal that involves clinical issues

Is there a process in place to monitor either the appeal and grievance process or the areas of concern
identified by enrollee appeals and grievances?

Resolution and notification: grievances and appeals (42 CFR 438.408 and 457.1260)
1.

Approximately how many grievances did the MCP receive in the most recent reporting year?

Approximately how many appeals did the MCP receive in the most recent reporting year?

Approximately what percent of notices of action on requests for service authorization or payment by
Medicaid and CHIP enrollees are appealed to the MCP?

Approximately what percent of notices of action on requests for service authorization or payment by
Medicaid and CHIP enrollees are appealed to the state fair hearing process?
a.

Approximately what percent of these are overturned by the state?

Expedited resolution of appeals (42 CFR 438.410 and 457.1260)
1.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of an appeal? If so, please describe this process. What are the time frames for this process?

Are physicians allowed to request expedited appeals on behalf of an enrollee? How does the MCP protect
physicians who make such requests?

Information about the grievance system to providers and subcontractors (42 CFR 438.414 and 457.1260)
1.

Who in your MCP has responsibility for the functioning of the grievance process and the authority to require
corrective action?

Did your state Medicaid and CHIP agency develop or approve the description of your MCP’s grievance
system provided to Medicaid and CHIP providers? [Note: clarify if the state Medicaid and CHIP agency
developed or approved] If it approved your description, how is the state’s approval documented?

Recordkeeping and reporting requirements: grievances and appeals (42 CFR 438.416 and 457.1260)
1.

Where in your MCP are records on Medicaid and CHIP enrollee grievances and appeals kept?

178 | PROTOCOL THREE

MCP Information Systems Staff Interview
Instructions. This interview will assess the MCP’s information management function and how it supports the other
functions of the organization, such as planning and operations, quality assessment and improvement program
activities, care coordination, etc. This is also an opportunity to explore the extent to which the health information
needs of the entire MCP and provider network are measured, assessed, and improved.
The interview should include MCP staff responsible for health information systems issues at the MCP. It should
include those responsible for technology implementation, as well as staff that are responsible for the information
quality, information transmittal, information sharing, and information policy and procedure development and
implementation.
Information system capabilities. The interviewees should receive a copy of the MCP’s most recent Information
Systems Capability Assessment (ISCA) (see Appendix A) that has either been completed by an independent
organization reviewing the MCP or has been completed by the organization conducting this compliance review. 60 The
findings of the ISCA will serve as a guide to conducting this interview. During this interview, validate the information
provided about the MCP on the ISCA, explore any areas of concern, and gather missing or additional information for
use in evaluating standards compliance, paying particular attention to how data are defined and captured across the
MCP and how data transmission and integration takes place across the MCP. Questions and areas of discussion
should be based on the findings of the ISCA, and may include:
1.

Are the findings of the most recent assessment of the MCP’s information systems capacity reflective of your
own assessment of capabilities?

What are your information system’s strengths and weaknesses?
a.

What has the MCP done to address information system problem areas?

What information needs does your MCP have that are not currently met by your present information system?
a.

What has the MCP done to address these needs?

Is the data collected from network providers on services to enrollees subject to accuracy and timeliness
checks?

Please describe procedures used to screen all data, both internal and external, for completeness, logic, and
consistency.

How is enrollee-specific data and information made available when and where needed by the MCP’s
provider network?

Delivery network (42 CFR 438.206 and 457.1230(a))
1.
2.

How does your information system track services provided by and/or reimbursed to out-of-network
providers?
Describe the capabilities to routinely collect data on use of out-of-network providers (excluding Point of

PROTOCOL THREE | 179

Service-related use).
a.

Is data on use of out-of-network providers separately available for Medicaid and CHIP enrollees?

Assurances of adequate capacity and services (42 CFR 438.207(b) – (c) and 457.1230(b))
1.

Please describe any information system capabilities used to demonstrate to the Medicaid and CHIP agency
that the MCP offers an appropriate range of preventive, primary care, specialty services, and LTSS that is
adequate for the anticipated number of enrollees in your service area.

Please describe any information system capabilities used to demonstrate to the Medicaid and CHIP agency
that the MCP maintains a network of providers that is sufficient in number, mix, and geographic distribution
to meet the needs of the anticipated number of enrollees in the service area.

Coordination and continuity of care for all enrollees (42 CFR 438.208 and 457.1230(c))
1.

How does the MCPs information system integrate medical, behavioral and LTSS assessments, care
planning, service planning and authorization information and processes?

Health information systems (42 CFR 438.242 and 457.1233(d))
1.

How is the data collected from network providers on services to enrollees checked for accuracy and
timeliness?

Please describe procedures used to screen all data, both internal and external, for completeness, logic, and
consistency.

180 | PROTOCOL THREE

Quality Assessment and Performance Improvement Program Staff Interview
Instructions. This interview with quality improvement program leaders and staff provides an opportunity to gain a
more thorough understanding of the approaches and processes used by the MCP to assess and improve quality.
Availability of services (42 CFR 438.206 and 457.1230(a))
1.

What information is generated through QAPI activities to assess the MCP’s availability of services?
a.

What issues were considered in the assessment process?

What services, such as family planning and women’s health services, have QAPI activities focused on?

Please describe the assessment results.
a.

Are there any service-specific results? If so, please describe them.

Has the MCP implemented QAPI findings relevant to the availability of services? If so, please describe them
and their results.

How frequently does the MCP evaluate the volume and enrollee access to LTSS services? What factors are
used in evaluation of the LTSS network? Note that this is not applicable to CHIP.

Furnishing of services-timely access (42 CFR 438.206(c) and 457.1230(a))
1.

Please describe any recent QAPI activities implemented to monitor the MCP’s compliance with its
established standards for timeliness of access to care and member services.
a.

Please describe any recent QAPI activities implemented to promote cultural competency and delivery of
services in a culturally competent manner.
a.

What were the results of these QAPI activities?

What are the results of these QAPI activities?

Please describe any recent QAPI activities implemented to promote physical access, reasonable
accommodations, and accessible equipment for Medicaid enrollees with physical or mental disabilities.
a.

What are the results of these QAPI activities?

Enrollee rights (42 CFR 438.100 and 457.1220)
1.

How is the enrollee’s right to be free from restraint or seclusion monitored for enrollees, including, for
example, those receiving LTSS? Note that requirements applying to LTSS are not applicable to CHIP.

Provider selection (42 CFR 438.214 and 457.1233(a))
1.

What type of information is generated through the quality improvement program to support re-credentialing
of individual practitioner providers?

What types of information does the quality improvement program provide to support the re-credentialing of
institutional and other non-practitioner providers?

PROTOCOL THREE | 181

What types of information does the quality improvement program provide to support the evaluation of LTSS
provider qualifications?

Practice guidelines (42 CFR 438.236 and 457.1233(c))
1.

Through what process does your MCP ensure necessary communication occurs between those responsible
for the QAPI program and the administrative function responsible for adopting practice guidelines?

Quality assessment and performance improvement program (42 CFR 438.330 and 457.1240(b))
1.

Does the state require the MCP to address a specific topic or topics and/or indicators in your performance
improvement projects? If yes, what types of projects are required?

How does the MCP define enrollees with “special health care needs”? How are these enrollees identified/
tracked within your MCP?

How does the MCP assess the quality and appropriateness of care including LTSS, furnished to enrollees
with special health care needs? Please provide examples.

Does the MCP evaluate the effectiveness of its quality assessment and performance improvement program?
How often?

Please describe the evaluation process. What aspects of the program are included in the evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of these findings?

Does the state require the MCP to evaluate the impact and effectiveness of its quality assessment and
performance improvement program?
a.

How does the MCP conduct its evaluation? What aspects of the program are included in the
evaluation?

How often does the MCP conduct its evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of these findings?

What is reported to the state, and how often?

What is the MCP’s role in the state’s evaluation?

What information, if any, does the MCP provide to the state?

182 | PROTOCOL THREE

What feedback, if any, does the MCP receive from the state? How does your MCP implement the
feedback?

What evaluation findings are reported to the state and how often?

What interventions are used or are anticipated to be used to improve LTSS quality? How will the
interventions be evaluated for effectiveness? How will improvement be sustained or increased?

Health information systems (42 CFR 438.242 and 457.1233(d))
1.

How are enrollee and provider data from all components of your MCP’s network used in your MCP’s quality
assessment and performance improvement program?
a.

Are there any components in your network for which you do not have adequate enrollee utilization and
provider data?

How is data obtained from the meaningful use of certified electronic health records (EHRs) utilized as part of
the MCP’s quality improvement program?

Handling of grievances and appeals (42 CFR 438.406 and 457.1260)
1.

What is the process used to monitor the appeal and grievance process?

What is the process to monitor areas of concern identified by enrollee appeals and grievances?

Recordkeeping and reporting requirements on grievances and appeals (42 CFR 438.416 and 457.1260)
1.

To what extent is information on Medicaid and CHIP enrollee grievances and appeals analyzed and included
as part of your MCP’s Quality Assessment and Performance Improvement Program?

PROTOCOL THREE | 183

Provider/Contractor Services Staff Interview
Instructions. This is an interview of MCP staff members who are responsible for establishing and maintaining
communications with the MCP’s individual practitioners and other types of health care providers (e.g., organizations).
This includes staff responsible for management of the credentialing process and oversight of delegated activities.
Through these interviews, the reviewer(s) will assess enrollee rights; the credentialing and appointment process;
oversight of the providers; and how information is communicated to providers.
Enrollee rights (42 CFR 438.100 and 457.1220 and 42 CFR 438.206-210 and 457.1230(a-d)))
1.

How does the MCP inform its providers (individual, institutional, and LTSS providers) about enrollee rights
and responsibilities?
a.

How does the MCP monitor for compliance with these rights by its providers?

To what extent, if any, does the MCP supply providers with information on where to refer enrollees who are
having difficulty understanding the materials that have been provided to them by the MCP?

Does the MCP require providers to have access to oral interpreter services?
a.

Does the MCP supply providers with guidance or assistance in accessing oral interpreter services if
necessary?

How does the MCP ensure that its own facilities and those of its affiliated providers comply with enrollee
rights to treatment with respect, dignity, and consideration for privacy?
a.

Are there any additional considerations made for providers of LTSS, where services may be of a more
intimate nature or occur in a more isolated setting? Please provide examples.

How does the MCP ensure that enrollees are not discriminated against in its own facilities and those of its
affiliated providers when seeking health care services consistent with their covered benefits?

Please describe the MCP’s credentialing, verification and oversight process for primary care providers, other
health care professionals, LTSS and institutional providers.
a.

What is encompassed by reviews and evaluations of these providers?

Do these processes involve visits to the providers’ care delivery sites?

What methods are used to encourage providers to share information on available treatment options and
alternatives with enrollees?

What processes are in place for monitoring providers to determine that they are providing information on
available treatment options and alternatives?

What requirements does the MCP have for providers/contractors relative to enrollee advance directives?
a.

How is it determined that providers/contractors are meeting the MCP’s requirements?

10. How does the MCP inform all of its network providers, including its LTSS, individual and institutional
providers, about enrollee rights to service availability, coordination and continuity of care, coverage and
authorization of service, and to obtain a second opinion from an appropriately qualified health professional?
a.

How does the MCP monitor for compliance with these rights by its providers?

184 | PROTOCOL THREE

11. How are the MCP’s network providers informed of enrollees’ right to request and receive a copy of their
medical records, and to request that they be amended or corrected?
12. What steps does the MCP take to ensure that providers/contractors are aware of and in compliance with
applicable federal and state laws on enrollee rights?
13. If a provider/contractor is found in violation of a federal or state law concerning enrollee rights, what action is
taken by the MCP?
Availability of services (42 CFR 438.206 and 457.1230(a))
1.

Please describe the MCP credentialing and re-credentialing process.
a.

Is the process different for Medicaid and CHIP providers than for providers serving other networks? If
yes, what are the differences?

How is it determined that providers are geographically accessible to Medicaid and CHIP enrollees and
physically accessible to enrollees with disabilities?

Please describe the processes for monitoring the provider network to determine that Medicaid and CHIP
requirements about timeliness, availability, and accessibility are being met.
a.

What are the most recent findings from this process?

How often in the last year has your MCP had to arrange for services or reimbursements to out-of-network
providers?

How does the MCP evaluate the expected utilization of institutional care in comparison with the use of
HCBS as an alternative?

Does the MCP maintain accessibility information on its LTSS and other specialized providers? If yes, how is
this maintained and shared with enrollees?

How does the MCP encourage the promotion of culturally competent service delivery by LTSS and other
specialty providers?

Are there any limits to choice of LTSS and other specialty providers?

Timely access to service (42 CFR 438.206(c)(1) and 457.1230(a))
1.

Ask only if MCP is a MCO, PIHP, or PAHP: Are your MCP’s provider services available 24 hours a day, 7
days a week, when medically or otherwise necessary to meet the enrollee’s needs?
a.

Are certain LTSS or other specialized providers/provider types contracted specifically for afterhours/urgent/emergent need?

If yes, what types? How were these types determined?

Are providers included in developing beneficiary emergency back-up plans? If they are not involved in the
back-up plan development, how are they made aware of their responsibility for emergency back-up?

Are the hours of operation of the provider network serving Medicaid and CHIP enrollees different from the
hours of operation of the provider network serving other enrollees? If yes, why are they different?

PROTOCOL THREE | 185

Does the MCP continuously monitor its provider network for compliance with established standards on
timeliness of access to all care and member services? If yes, how, and what are the most recent findings?

What steps are taken to address provider non-compliance with established standards for timeliness of
access to care and member services?
a.

How are corrective actions assessed for effectiveness? Please describe the follow up and monitoring.

Coordination and continuity of care for all enrollees (42 CFR 438.208 and 457.1230(c))
1.

How are primary care providers serving enrollees with special health care needs made aware of and involved
in procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Assuring appropriate use of specialists?

Coordinating primary care services with care provided by other MCOs, PIHPs or PAHPs serving the
enrollee?

Coordinating care with other providers, including specialist and LTSS providers?

Additional coordination and continuity of care questions: SHCN (42 CFR 438.208 and 457.1230(c))
1.

How are specialty providers serving enrollees with special health care needs made aware of and involved in
procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Coordinating specialty care services with care provided by other MCPs serving the enrollee?

Coordinating care with other providers, including primary and LTSS providers?

How are LTSS providers serving enrollees with special health care needs made aware of and involved in
procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Coordinating care with other providers, including primary and LTSS providers?

Additional coordination and continuity of care questions: LTSS (42 CFR 438.208
1.

How are LTSS providers serving enrollees with special health care needs made aware of and involved in
procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

186 | PROTOCOL THREE

Coordinating care with primary care and specialty providers?

Coverage and authorization of services (42 CFR 438.210 and 457.1230(d))
1.

Do contracts/agreements with individuals or organizations performing utilization review offer any performance
incentives? If yes, please describe the incentives. [Note to reviewers: Look for any incentives for denying,
limiting, or discontinuing authorization of services.]

Are network providers notified of the information ordinarily required to process an authorization request?

Please describe the process for notifying the requesting provider of any decision to deny, limit, or discontinue
authorization of services.
a. What are the MCP’s time frames for notification?

Does the MCP contract with all LTSS and other specialized provider types identified in the state’s benefit
package? If not, what provider types are not contracted? How are enrollees’ needs met in lieu of this service
availability?

Are there any universal service limitations on LTSS? If yes, what are the service limitations, and how were
these determined?

Provider selection (42 CFR 438.214 and 457.1233(a))
1.

What types of individual practitioners are subject to the MCP’s credentialing process?

Please describe the MCP’s credentialing processes for individual practitioners.
a. How often does this process take place?
b. What items of credentials information are updated during the process?
c. Are site visits made to providers? When and how often? How is it determined that a site visit will be
made?
d. Who is involved in the MCP’s credentialing activities?

Please describe the MCP’s re-credentialing processes for individual practitioners.
a. What types of information are monitored and reviewed during the re-credentialing process?
b. What other operations of the MCP contribute information to be used in the re-credentialing process?

Ask only if MCP is a MCO, PIHP, or PAHP: Please describe the MCO’s /PIHP’s/PAHP’s processes for
selecting and monitoring institutional and other non- practitioner network providers (including LTSS).
a. What information is reviewed as a part of this process?
b. Are site visits made? When and how often?

Please describe the MCP’s credentialing and re-credentialing processes for institutional providers.
a. Are site visits a part of the process to credential and re-credential institutional providers?

PROTOCOL THREE | 187

b. How frequently is re-credentialing performed?
c. What items of information are typically reviewed during the evaluation and reevaluation process?
6.

What other MCP operations contribute to the evaluation of a network institutional provider?

What criteria is the basis for denial of provider participation in the MCP’s network?

How does the MCP verify the skills and requirements of LTSS providers, including self-directed support
options? (i.e., background checks, exclusions, certifications and/or licensures)

Grievance systems (42 CFR 438.228)
1.

Please describe the process for notifying the requesting provider of any decision to deny, limit, or discontinue
authorization of services.
a. What are the MCP’s time frames for notification?

Sub contractual relationships and delegation (42 CFR 438.230 and 457.1233(b))
1.

What types of activities are performed by (and thereby delegated to) contractors?

Please describe your MCP’s process for identifying and selecting contractors.
a.

What steps does your MCP take to determine that an entity to which functions will be delegated is capable of
performing the delegated functions?
a.

How is it determined that a contractor has the ability to perform the activities that are being delegated by
the MCP?

Please describe any evaluation process that your MCP has in place.

For each of the activities that have been delegated:
a. Is there any ongoing monitoring and review of entities performing delegated activities?
i.

How this is accomplished?

ii.

Is the process the same for all delegates at all times?

iii.

Are there any instances when your MCP varies the monitoring process or the timing of
evaluation?

b. Does your MCP perform an annual evaluation of the delegate’s sub-contractor’s performance?
i.

Please describe the process to conduct this evaluation. What is included in the evaluation?

c. What is done with the results of delegate evaluations?
i.

Do the results of the most recent delegate subcontractor evaluations specify any necessary
corrective action for problems or deficiencies identified?

ii.

Please describe some of the recommendations made to delegates in an effort to improve
performance.

188 | PROTOCOL THREE

What steps does your MCP take to assure that the delegate implements corrective actions?

Who in the MCP is assigned responsibility for monitoring the delegate’s performance?

Does the MCP delegate any of its activities to LTSS providers? If yes, how is the provider’s ability to carry
out delegated activities determined and monitored?

Practice guidelines (42 CFR 438.236 and 457.1233(c))
1.

What mechanism is in place to consult affiliated providers as practice guidelines are adopted and reevaluated?

How are practice guidelines disseminated to providers?

Quality assessment and performance improvement program (42 CFR 438.330)
1.

How does the MCP monitor LTSS provider quality, appropriateness of care, compliance with state and plan
requirements, and enforce corrective action when necessary?

Please describe any QAPI activities implemented to assess or improve communications with the MCP’s
providers.
a.

What are the results of these activities?

Please describe any QAPI activities implemented to assess or improve the credentialing process and
oversight of the MCP’s delegated activities.
a.

What are the results of these activities?

Health information systems (42 CFR 438.242 and 457.1233(d))
1.

Does the MCP have data collection requirements for LTSS providers, health care facilities, and physicians?
a.

How are the requirements communicated to these organizations and individuals?

If issues arise in the timeliness and accuracy of the data that is being collected and submitted, who notifies
the health care facility or physician?

Information about the grievance system to providers and subcontractors (42 CFR 438.414 and 457.1260)
1.

When are providers given information about the MCP’s Medicaid and CHIP complaint and grievance
system?
a.

What is typically included in the information given to providers relative to Medicaid and CHIP
grievances?

PROTOCOL THREE | 189

Enrollee Services Staff Interview
Instructions. The enrollee services staff interview provides an opportunity to speak with MCP staff members who are
responsible for communicating with enrollees. Relevant staff includes those individuals responsible for written
communication, phone responses to inquiries and problems, the complaint and grievance system and other services
designed to assist Medicaid and CHIP enrollees in their use of MCP services. Through this interview, the EQRO will
assess the manner in which the MCP and its provider network address issues relating to the rights of enrollees; the
MCP’s efforts regarding enrollee education and communication; the mechanisms in place to insure that information
needed to provide services to enrollees is available throughout the MCP; and the aspects of enrollee services are
measured, how collected data is assessed, and what efforts have been made to improve enrollee services.
Enrollee right to information (42 CFR 438.100 and 457.1220; 42 CFR 438.10 and 457.1207)
1.

What information is routinely provided to Medicaid and CHIP enrollees?
a.

What is the process for disseminating information to new and existing enrollees?

How often is information distributed to existing enrollees?

In what format is this information presented?

Please describe or provide copies of the formats in which information is presented to enrollees.

In what languages or alternative formats are enrollee materials and information presented? If yes, how was
it determined that materials were needed in different languages?

Does the MCP provide written materials in alternative formats for the visually impaired? If yes, how did the
MCP determine that materials were needed for the visually impaired?

Please describe the procedure for handling calls to the MCP from non-English speaking enrollees.
a.

What instruction or guidance is available for providers that may need interpretation assistance to
provide care and services to assigned enrollees?

To what extent is the MCP responsible for responding to requests for information for potential Medicaid and
CHIP enrollees?

How does the MCP inform enrollees (and potential enrollees, if applicable) about how to obtain oral
interpreter services if they have limited proficiency in English?

Are there any benefits that an enrollee is entitled to under the Medicaid and CHIP program, including LTSS
benefits, but that are not made available through the MCP contract? If yes, what are those benefits? How
are enrollees made aware of the Medicaid and CHIP program benefits that are outside the scope of services
available through the MCP?

How does the MCP ascertain the primary language spoken by the individual Medicaid and CHIP enrollees?

10. Are enrollees provided with a listing of primary care providers? If yes, does the listing include providers’
non-English language capabilities?
11. Does your MCP give written notice of termination of a contracted provider to enrollees who receive primary
care from, or are seen on a regular basis by, the terminated providers? If yes, how is this accomplished?
Have you had to make any such notifications in the last year?

190 | PROTOCOL THREE

12. Does your MCP give enrollees any notice of significant changes in the information in the Enrollee
Handbook? When and how does this occur? Have you had to make any such notifications in the last year?
13. How does the MCP ensure that information and instructional materials intended for enrollees and potential
enrollees are easily understood by those with a variety of cognitive and intellectual capabilities?
14. How does the MCP provide its enrollees information about provider appeal rights regarding coverage of a
service?
Enrollee right to respect, dignity, and privacy (42 CFR 438.100 and 457.1220; and 42 CFR 438.206-210 and
457.1230(a-d))
1.

How does the MCP ensure that its own facilities and those of its affiliated providers comply with enrollee
rights to treatment with respect, dignity, and consideration for privacy and confidentiality of information?
a.

Are there any additional considerations made for providers of LTSS or other specialized services, where
services may be of a more intimate nature or occur in a more isolated setting? Please provide
examples.

Enrollee right to participate in decisions regarding his or her health care (42 CFR 438.100 and 457.1220); and
regarding advance directives (42 CFR 438.10(g) and 457.1207; and 42 CFR 438.206-210 and 457.1230(d))
1.

To what extent does the MCP allow enrollees to participate in care and treatment decisions? Please
describe some of the ways in which this is accomplished.

To what extent are Medicaid and CHIP enrollees informed at the time of enrollment of their right to accept or
refuse treatment and to execute an advance directive, and the MCP’s policies on implementation of that
right?

Enrollee right to service availability, coordination and continuity of care, coverage and authorization of
service, and to obtain a second opinion from an appropriately qualified health professional (42 CFR 438.100
and 457.1220; and 42 CFR 438.206-210 and 457.1230(a-d))
1.

How does the MCP monitor compliance of enrollee rights to service availability, coordination and continuity
of care, coverage and authorization of service, and to obtain a second opinion from an appropriately
qualified health professional?
a.

What are the most recent results of this monitoring?

Enrollee right to request and receive medical records (42 CFR 438.100 and 457.1220; and 42 CFR 438.206-210
and 457.1230(a-d))
1.

How do enrollees obtain access to their medical records maintained by the MCP, including records
maintained by providers/contractors from whom the enrollee has received services?

How are enrollees informed of their right to request and receive a copy of their medical records, and to
request that they be amended or corrected?

Has the MCP received any complaints about an enrollee’s inability to access their medical records in a
timely manner? If yes, what was the volume and nature of the complaints? How were they resolved?

PROTOCOL THREE | 191

Compliance with other federal and state laws (42 CFR 438.100 and 457.1220; and 42 CFR 438.206-210 and
457.1230(a-d))
1.

Does the MCP orient staff to the federal and state laws on enrollee rights that must be observed during dayto-day operations? Does the MCP remind staff of the importance of observing these laws during
interactions with other employees and with enrollees?

Describe the procedure for handling an enrollee complaint involving a perceived violation of their rights.

Availability of services (42 CFR 438.206 and 457.1230(a))
1.

What processes does the MCP take to monitor availability and accessibility of services to Medicaid and
CHIP enrollees?
a.

What are the most recent findings from this process?

Is there any information that is routinely collected and monitored to determine that care and services are
being rendered to Medicaid and CHIP enrollees in a timely manner?
a.

What are the most recent findings of this monitoring?

Availability of services-delivery network (42 CFR 438.206(b) and 457.1230(a))
1.

Are Medicaid and CHIP enrollee requests for out-of-network providers tracked?
a.

How often do Medicaid and CHIP enrollees request services from out-of-network providers?

What are their reasons for requesting out-of-network providers?

How often do Medicaid and CHIP enrollees receive services from out-of-network providers?

Availability of services-Furnishing of services (42 CFR 438.206(c) and 457.1230(a))
1.

Ask only if MCP is a MCO/ PIHP or PAHP: Are MCO/ PIHP/PAHP and provider services available 24
hours a day, 7 days a week, when medically appropriate?

How frequently does enrollee services staff receive complaints about provider hours of operation not being
available to enrollees when medically necessary?

Does the MCP conduct surveys, focus groups or other activities to receive the feedback of Medicaid and
CHIP enrollees? If yes, what are the most recent findings about Medicaid and CHIP enrollee perceptions
about availability of MCP and provider services?

Coordination and continuity of care (42 CFR 438.208 and 457.1230(c))
1.

How are Medicaid and CHIP enrollees with special health care needs—including both individuals with
special health care needs identified by your MCP and individuals identified by the state Medicaid and CHIP
agency or its agent

How does this MCP identify and assess Medicaid and CHIP enrollees with special health care needs?

What proportion of Medicaid and CHIP enrollees has an ongoing source of primary care?

192 | PROTOCOL THREE

Coverage and authorization of services (42 CFR 438.210 and 457.1230(d))
1.

How frequently does enrollee services staff receive complaints about difficulty obtaining emergency or poststabilization services?

Please describe the procedure for handling member calls regarding need for emergency services.

Enrollment and disenrollment (42 CFR 438.56 and 457.1212)
1.

Please describe the procedures that are followed when a request for disenrollment is received from an
enrollee.

How is disenrollment information tracked through or by other MCP operations (e.g., grievance process,
quality improvement, administration)?
a.

How many requests by Medicaid and CHIP enrollees were received last year for disenrollment?

What were the cited causes?

Grievance systems (42 CFR 438.228)
1.

Please describe the process for notifying Medicaid and CHIP enrollees of any decision to deny, limit, or
discontinue a request for service.
a.

What are the MCP’s time frames for notification?

Practice guidelines (42 CFR 438.236 and 457.1233(c))
1.

How often does your MCP receive requests from enrollees and potential enrollees for practice guidelines?
How does your MCP respond to these requests?

When and how does your MCP disseminate practice guidelines to enrollees?

Grievance system - general requirements (42 CFR 438.402 and 457.1260)
1.

What enrollee materials contain information about the complaint and grievance processes? When are
enrollees presented with this information?

Please describe the process for handling authorization decisions that are adverse to the enrollee.

Handling of grievances and appeals (42 CFR 438.406 and 457.1260)
1.

What MCP department or staff members are responsible for assisting enrollees to use the organization’s
complaint or grievance system, including completing forms, or taking other steps to resolve an appeal or
grievance? What kind of assistance is made available to Medicaid and CHIP enrollees?

What are the qualifications and credentials of individuals who make decisions on grievances and appeals?
a.

How does the MCP ensure that these individuals have not been involved in any previous level of
review or decision-making?

How does the MCP ensure that these individuals have the appropriate clinical expertise in treating the
enrollee’s condition or disease, if deciding any of the following:

PROTOCOL THREE | 193

An appeal of a denial that is based on lack of medical necessity

ii.

A grievance regarding denial of expedited resolution of an appeal

iii.

A grievance or appeal that involves clinical issues

How does your MCP treat oral requests by Medicaid and CHIP enrollees to appeal actions?

As part of an appeal, to what extent do enrollees and their representatives have an opportunity to:
a.

Present evidence, and

Examine the enrollee’s case file, including medical records, and any other documents and records
considered during the appeals process

Resolution and notification: Grievances and appeals (42 CFR 438.408 and 457.1260)
1.

Please describe the MCP’s grievance resolution process.

Please describe the MCP’s appeal resolution process.

How is it determined that an enrollee’s appeal requires expedited resolution?

What percent of appeal resolutions that are completely or partially adverse to Medicaid and CHIP enrollees
are appealed to the state fair hearing process or review? Of these, what percent are overturned by the state
Medicaid and CHIP agency?

Expedited resolution of appeals (42 CFR 438.410 and 457.1260)
1.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of an appeal? If yes, please describe this process.
a.

What are the time frames defined for this process?

How does the MCP notify enrollees of any denials of a request for expedited resolution?

Have there been any complaints by Medicaid and CHIP enrollees that their requests for expedited appeals
have not been acted upon timely (e.g., within three working days). If yes, how many such complaints were
received in the year under review?

Record keeping and reporting requirements (42 CFR 438.416 and 457.1260)
1.

How are Medicaid and CHIP grievances and appeals registered and tracked for resolution? Is each
grievance and appeal tracked through to resolution?

How often is Medicaid and CHIP grievance and appeal information analyzed for trends?

Who receives this analysis?

Does the MCP provide any information to the state relative to its grievances and appeals?

How long are Medicaid and CHIP grievance and appeal records retained?

194 | PROTOCOL THREE

To what extent is information on Medicaid and CHIP enrollee grievances and appeals analyzed and included
as part of your MCP’s Quality Assessment and Performance Improvement Program?

Continuation of benefits while the MCP or PIHP appeal and the state fair hearing are pending (42 CFR
438.420)
1.

What happens to enrollee benefits once continuation of benefits has been denied by the MCP, and an
appeal has been filed by the enrollee or the treating physician?
a.

Are there any mechanisms in place to continue the benefits pending the outcome of the appeal? If yes,
under what circumstances?

PROTOCOL THREE | 195

Utilization Management Staff Interview
Instructions. MCP interview participants should include the Medical Director, utilization management directors or
managers, utilization management review staff, case managers or care coordinators, and any other individuals who
have information pertinent to these regulatory provisions. [Note: This interview can be combined with the Medical
Director interview or the Care Coordinators and Case Managers interview.]
The utilization management interview provides an opportunity to discuss with the MCP staff responsible for tracking
and managing the utilization of MCP services. Through these interviews, the reviewer(s) will assess delivery network,
service authorization; the use of practice guidelines, and grievances and appeals; and management of resources
across all MCP network provider sites where enrollees receive health care.
Availability of services (42 CFR 438.206)
1.

How frequently does the MCP evaluate the volume and enrollee access to LTSS services? What factors are
used in evaluation of the LTSS network?
a.

How does the MCP evaluate the expected utilization of institutional care in comparison with use of
home and community based services (HCBS) as an alternative?

How frequently does the MCP evaluate the volume of and enrollee access to family plan and women’s
health services? What factors are sued to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to specialist health services?
What factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to children’s dental care? What
factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to behavioral health services?
What factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to family planning and women’s
health services? What factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to any other specific services,
such as HIV and foster care services? What factors are used to evaluate the network?

Delivery network (42 CFR 438.206(b) and 457.1230(a))
1.

What procedures must a Medicaid and CHIP enrollee follow if he/she wishes to receive a second opinion?
a.

For what types of services are second opinions available?

Coverage and authorization of services (42 CFR 438.210 and 457.1230(d))
1.

What types of services require pre-authorization?

What are the MCP’s time frames for processing standard and expedited requests for service authorization?

How does the MCP monitor its compliance with these time frames?
a.

What sources of documentation exist to provide evidence of the monitoring by the MCP?

196 | PROTOCOL THREE

How often and under what circumstances are requesting providers consulted when the MCP makes service
authorization decisions?

To what extent does the MCP assess the consistency of authorization decisions? How does the MCP do
this?

What is the process when a decision is being made to deny authorization for a service?
a.

Who makes the decision to deny a request to authorize a service?

Please describe the process for notifying the requesting provider and the enrollee of any decision to deny,
limit, or discontinue authorization of services.
a.

What information is typically included in enrollee and provider notification?

What are the MCP’s time frames for notification?

To what extent if at all s inappropriate use of emergency rooms by your Medicaid and CHIP enrollees a
concern for your MCP?

Has your MCP investigated a potential relationship between inappropriate emergency department use and
enrollee access to routine and urgent care, or reviewed the most frequent diagnosis resulting in
inappropriate emergency department use?

10. What was the volume of denied claims for emergency and post-stabilization services in the most recent
year?
11. Does the authorization process differ between acute and primary services and LTSS, or any other
providers? If yes, how?
Grievance systems (42 CFR 438.228)
1.

What types of services require pre-authorization?

Please describe the process for notifying the requesting provider and the enrollee of any decision to deny,
limit, or discontinue authorization of services.
a.

What information is typically included in enrollee and provider notification?

What are the MCP’s time frames for notification?

How does your MCP track requests for covered services that the MCP or its providers has denied?

What was the volume of denied request for services in the most recent year?

Application of practice guidelines (42 CFR 438.236(c) and 457.1233(c))
1.

What practice guidelines have the MCP adopted?

To what extent are your utilization management review guidelines (criteria) consistent with these practice
guidelines?
a.

How do you promote or ensure consistency?

PROTOCOL THREE | 197

Please describe how utilization management review guidelines (criteria) are modified to reflect the adoption
or revision of practice guidelines.
a.

Are both sets of guidelines updated through the same process, at the same time?

Quality assessment and performance improvement program (42 CFR 438.330 and 457.1240(b))
1.

What information is analyzed to detect over- and under-utilization of services?
a.

Who is involved in the analysis and review of this information?

What, if any trends been identified?

What are the typical follow-up actions taken when either condition is discovered?

How does the MCP monitor LTSS utilization patterns? Are there any services for which specialized or
more focused utilization analysis is used?

Grievance system - General requirements (42 CFR 438.402 and 457.1260)
1.

Please describe the appeals process and the role of utilization management staff in the resolution process.
Elaborate on the particular steps, including time frames, in which utilization management staff is involved.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of an appeal? Please describe this process and its time frame.

Does the MCP’s grievance and appeal system differ for LTSS vs. acute and primary care services? If yes,
how?

Handling of grievances and appeals (42 CFR 438.406 and 457.1260)
1.

What MCP department or staff is responsible for assisting enrollees in using the MCP’s appeal or grievance
system, including completing forms, or taking other steps to resolve an appeal or grievance?

What are the qualifications and credentials of individuals who make decisions on grievances and appeals?
a.

How does the MCP ensure that these individuals have not been involved in any previous level of
review or decision-making?

How does the MCP ensure that these individuals have the appropriate clinical expertise in treating the
enrollee’s condition or disease, if deciding any of the following:
i.

An appeal of a denial that is based on lack of medical necessity

ii.

A grievance regarding denial of expedited resolution of an appeal

iii.

A grievance or appeal that involves clinical issues

Expedited resolution of appeals (42 CFR 438.410 and 457.1260)
1.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of a grievance? If yes, please describe this process. What are the time frames defined for this
process?

198 | PROTOCOL THREE

How does the MCP notify enrollees of any denials of a request for expedited resolution?

Continuation of benefits while the MCP or PIHP appeal and the state Fair Hearing are pending (42 CFR
438.420)
1.

What happens to enrollee benefits once continuation of benefits has been denied by the MCP, and an
appeal has been filed by the enrollee or the treating physician or other provider, including providers of
LTSS?
a.

Are there any mechanisms in place to continue the benefits pending the outcome of the appeal and if
so, under what circumstances?

How are enrollees notified of this mechanism?

PROTOCOL THREE | 199

Medical Directors Interview
Instructions. The interview with the Medical Director provides an opportunity to assess MCP processes for
authorizing services and coverage for those services. The interview will address such topics as provider involvement
in the review of criteria used in the utilization management process, consistency between utilization management
criteria and practice guidelines, and Quality Assessment and Performance Improvement efforts.
Coverage and authorization of services (42 CFR 438.210 and 457.1230(d))
1.

How does the MCP monitor its compliance with the state’s time frames for processing standard requests for
service authorization?

What are the MCP’s standards for processing expedited requests for service authorization? How does the
MCP monitor its compliance with these time frames?

Under what circumstances are requesting providers consulted when responding to service authorization
requests?

How does the MCP ensure consistent application of criteria used in making service authorization decisions?

What mechanism does the MCP use to assure that any decision to deny a service authorization request or
to authorize a service in an amount, duration or scope that is less than requested, be made by a health care
professional who has appropriate clinical expertise in treating the enrollees’ condition or disease or by a
professional with expertise in serving special populations (e.g. Developmental Disabilities), in special
services (e.g., Vocational Rehabilitation), or with other LTSS expertise as appropriate?

How are employees and any contractors used by the MCP to perform service authorization and utilization
management financially compensated?

Are they paid in any way other than on a straight salary or per case review basis?

Do their financial compensation arrangements involve the use of any financial incentives?

How does the MCP apply the definition of ‘medically necessary services’ to LTSS for activities that support
age-appropriate growth and development and/or the ability to attain, maintain or regain functional capacity?

Quality assessment and performance improvement program (42 CFR 438.330 and 457.1240(b))
1.

Does the MCP have any processes for reviewing claims, payment systems, encounter data, electronic
health records, and medical records to assess utilization of services?
a.

Does the MCP utilize a health information exchange process?

What reports on service utilization are regularly produced by these processes?

What are the most recent findings with respect to over- and under-utilization?

How does your MCP define enrollees with “special health care needs”? How are these enrollees identified
within your MCP?

How does your MCP assess the quality and appropriateness of care furnished to enrollees with special
health care needs? Please provide examples.

200 | PROTOCOL THREE

Does the state require your MCP to address a specific topic or topics in your performance improvement
projects? If yes, what types of projects are required? For each PIP, at a minimum, include how significant
improvement was measured, how improvement will be/was sustained, and how beneficiary health outcomes
and satisfaction will be/was measured, and how the intervention will/has improved access and/or quality of
care.
a.

For duals-only MCPs, was a Medicare Advantage PIP substituted for a state-required PIP?

If CMS specifies any performance measures, what performance measures are collected and submitted,
if any?

Does the state require your MCP to evaluate the impact and effectiveness of its quality assessment and
performance improvement program?
a.

How does your MCP conduct its evaluation? What aspects of the program are included in the
evaluation?

How often does your MCP conduct its evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of these findings?

What is reported to the state, and how often?

What is your MCP’s role in the state’s evaluation?

What information, if any, does your MCP provide to the state?

What feedback, if any, does your MCP receive from the state? How does your MCP implement the
feedback?

PROTOCOL THREE | 201

Case Managers and Care Coordinators Interview
Instructions. Case managers and care coordinators typically are among the few MCP staff with opportunity to
interact closely and directly with Medicaid and CHIP enrollees. These individuals are often responsible for guiding
enrollees to the care and services available through their benefits and the provider network. These individuals play a
key role in assisting enrollees in managing and maintaining their health and managing complex conditions.
Interviewing these individuals will provide reviewers the opportunity to discuss topics surrounding MCP processes
related to service availability, enrollee needs and special populations, and continuity and coordination of care. [Note:
This interview can be combined with the Medical Director interview or the Utilization Management interview.]
Enrollee rights (42 CFR 438.100, and 42 CFR 438.206-210)
1.

How are the available options for LTSS identified and presented to enrollees?

How are enrollees engaged in decisions about the use of LTSS?

How is the enrollee’s right to be free from restraint or seclusion monitored for enrollees receiving LTSS?

Enrollee right to participate in decisions regarding his or her health care (42 CFR 438.100(b)(iv) and 457.1220)
1.

To what extent does the MCP allow enrollees to participate in care and treatment decisions? Please
describe some of the ways in which this is accomplished.

Availability of services (42 CFR 438.206)
1.
2.

How does the MCP evaluate the expected utilization of institutional care in comparison with the use of
HCBS as an alternative?
How does the MCP evaluate availability of services for individuals with intellectual and developmental
disabilities? What factors are used to evaluate the network?

How does the MCP evaluate availability of services for children with special health care needs? What factors
are used to evaluate the network?

How does the MCP evaluate availability of services for individuals with behavioral health conditions? What
factors are used to evaluate the network?

How does the MCP evaluate availability of services for dual-eligibles? What factors are used to evaluate the
network?

How does the MCP evaluate availability of services for individuals with HIV? What factors are used to
evaluate the network?

What methods does the MCP use to improve cultural competency?

Furnishing of services and timely access (42 CFR 438.206(c) and 457.1230(a))
1.

To what extent are services offered through the MCP available to Medicaid and CHIP enrollees and others
coordinating care 24 hours per day, 7 days per week when medically necessary?

What types of services require pre-authorization?

202 | PROTOCOL THREE

Coordination and continuity of care (42 CFR 438.208 and 457.1230(c))
1.

Does this MCP screen Medicaid and CHIP enrollees to identify those with special health care needs? If yes,
how is this implemented?

How are Medicaid and CHIP enrollees with special health care needs—including any individuals with special
health care needs identified by your MCP and any identified by the state Medicaid agency or its agent—
identified and tracked within your MCP?

Does this MCP assess Medicaid and CHIP enrollees with special health care needs? If yes, how are these
activities conducted?

Does this MCP require written treatment plans to be developed for enrollees with ongoing special conditions
that require a course of treatment or regular care monitoring? If yes, how is it decided which Medicaid and
CHIP enrollees will receive a written treatment plan?

If treatment plans are required by this MCP, how does the MCP ensure that treatment plans for individuals
with special health care needs address the needs identified by the assessment?

Please describe the treatment planning process for individuals with special health care needs and the
process for determining and assuring appropriate use of specialists.

Within the last year, how many treatment plans have been developed?

How many requests made by enrollees for review of treatment plans have been denied?

What were the reasons for these denials?

How many treatment plans have been denied?

What were the reasons for these denials?

What process(es) is/are used to coordinate services for enrollees?
a.

Are their different types of care coordination mechanisms for different types of enrollees? If yes, how
are these different and how do they work?

Who is responsible for coordinating the care of individuals with special health care needs?

10. What are the procedures for coordinating the services that the MCP furnishes to the enrollee with services
the Medicaid and CHIP enrollee receives from any other MCOs, PIHPs, and PAHPs?
11. If the MCP establishes separate coordination of care for medical services, LTSS, and mental health and
substance abuse services, how does the MCP ensure exchange of necessary information between
providers?
12. How is post-acute care coordinated?
13. How are LTSS providers involved in person-centered assessment, person-centered care and service
planning, coordination and authorization processes?

PROTOCOL THREE | 203

Coverage and authorization (42 CFR 438.210 and 457.1230(d))
1.

What types of services require pre-authorization?

Are emergency back-up plans created for all enrollee’s? If not, how is the need for an emergency back-up
plan determined? How is the emergency back-up plan shared with all appropriate parties?

Quality assessment and performance improvement program (42 CFR 438.330 and 457.1240(b))
1.

What processes does the MCP have to detect underutilization and overutilization? What activities, such as
QAPI projects, has the MCP implemented to address these issues?
a.

What are the results of these activities?

What activities, such as QAPI projects, has the MCP implemented to assess and improve care
coordination?
a.

What are the results of these activities?

204 | PROTOCOL THREE

Providers and Contractors Interview (as appropriate and time and resources permit)
Instructions. While interviewing providers and contractors requires additional time and resources, it is an opportunity
to obtain further information about MCP performance from those health care and LTSS professionals and institutions
that often serve as the first point of contact for Medicaid and CHIP members and health care providers. Provider and
contractor interviews should therefore be considered as an optional component of this protocol, to be considered
whenever there is a strong need for additional information and when time and resources permit. The interview
participants should be selected from the provider network and should offer representative view of the breadth of the
MCP’s primary care, specialist, LTSS, and institutional providers. These persons can often clarify issues pertaining to
communication, traversing the system, assuring enrollee rights, and delivery of care and services to the enrolled
population.
There are several ways to conduct the interview. It can be arranged with a group of individual health care
practitioners, a group of institution representatives, and a group of LTSS providers, or coordinated as one interview
for each group or as a combined group. Geographic location of providers should be considered, and conference calls
are a viable option for conducting an interview of this type, and often preferred by providers as only a brief
interruption in their daily activities. For this interview to be effective, reviewers should emphasize that this is an
opportunity to provide insight on the MCP’s performance and not an evaluation of the care and services offered to
Medicaid and CHIP enrollees.
Enrollee rights (42 CFR 438.10 and 457.1207) and Enrollee information (42 CFR 438.100 and 457.1220; and 42
CFR 438.206-210 and 457.1230(a-d))
1.

When the MCP’s enrollees present for services, do they appear to have a clear understanding of their rights,
responsibilities, and benefits? How to obtain services?

Does the MCP provide you with information on where to refer enrollees who are having difficulty
understanding the materials that have been provided to them by the MCP?

How often do you and your staff have to assist enrollees with understanding the materials provided by the
MCP?

Does the MCP require providers to have access to oral interpreter services?

Does the MCP provide your office with guidance or assistance is accessing interpreter services if
necessary?

Enrollee rights to receive information on available treatment options (42 CFR 438.102 and 457.1222; and 42
CFR 438.206-210 and 457.1230(a-d)); Provider-enrollee communications (42 CFR 438.100 and 457.1220; and
42 CFR 438.206-210 and 457.1230(a-d))
1.

Does the MCP place any limits on your ability to counsel or advise a Medicaid and CHIP enrollee on
treatment options that may be appropriate for the enrollee’s condition or disease?

Does the MCP encourage providers to share with enrollees information on available treatment options and
alternatives?
a.

Does this include options and alternatives that are within as well as those outside the scope of the
enrollees benefits? If yes, how does the MCP do this?

Availability of Services: Furnishing of services (42 CFR 438.206(c) and 457.1230(a))
1.

Are your hours of operation for Medicaid and CHIP enrollees different from the hours of operation for other
MCP enrollees? If yes, why?

PROTOCOL THREE | 205

Practice guidelines (42 CFR 438.236 and 457.1233(c))
1.

Are affiliated providers/contractors consulted as practice guidelines are adopted and re-evaluated?

How does the MCP make providers/contractors aware of practice guidelines currently in use and those
under consideration for adoption?

Expedited resolution of appeals (42 CFR 438.410 and 457.1260)
1.

Have there been any instances in the most recent year under review when the MCP took any punitive action
against you for requesting an expedited resolution of an appeal on behalf of Medicaid and CHIP enrollees or
for supporting an enrollee’s appeal?

END OF WORKSHEETS FOR PROTOCOL 3

206 | PROTOCOL THREE

Protocol 4. Validation of Network
Adequacy
A Mandatory EQR-Related Activity

ACTIVITY 2: IDENTIFY DATA SOURCES FOR VALIDATION
ACTIVITY 3: REVIEW INFORMATION SYSTEMS UNDERLYING
NETWORK ADEQUACY MONITORING
ACTIVITY 4: VALIDATE NETWORK ADEQUACY MONITORING DATA,
METHODS, AND RESULTS
ACTIVITY 5: COMMUNICATE PRELIMINARY FINDINGS TO MANAGED
CARE PLANS
ACTIVITY 6: SUBMIT FINDINGS TO STATE

Background
States must ensure that Medicaid and Children’s Health Insurance Program
(CHIP) managed care plans (MCPs) maintain provider networks that are
sufficient to provide timely and accessible care to Medicaid and CHIP
beneficiaries across the continuum of services. As set forth in 42 CFR 438.68,
states are required to set quantitative network adequacy standards for MCPs that
account for regional factors and the needs of the state’s Medicaid and CHIP
populations. 61
The purpose of this protocol is to guide the external quality review organization
(EQRO) in conducting the validation of network adequacy during the preceding
12 months to comply with requirements set forth in 42 CFR 438.68 and, if the
state enrolls American Indians and Alaska Natives (AI/AN) in the MCO, PIHP,
or PAHP, 42 CFR 438.14(b)(1). 62 This includes validating data to determine
whether the network standards, as defined by the state, were met. It does not
include evaluating the state’s network adequacy standards. This mandatory
external quality review (EQR) protocol applies to managed care organizations

eCFR :: 42 CFR Part 438 -- Managed Care Network Adequacy at 42 CFR 438.68

eCFR :: 42 CFR Part 438 -- Managed Care EQR Activities at 42 CFR 438.358(b)(iii)

PROTOCOL FOUR

ACTIVITY 1: DEFINE THE SCOPE OF THE VALIDATION

207

(MCOs), prepaid inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs),
which are collectively referred to here as MCPs.
Regulations providing for provider-specific network adequacy standards are set forth at 42 CFR
438.68(b). The Medicaid network adequacy standards are applied to CHIP per 42 CFR
457.1218. 63 See Box 4.1 for network adequacy provisions in the managed care final rule.
Box 4.1. Network Adequacy Provisions Under 42 CFR 435.68(b) and 457.1218
• States that contract with MCPs to provide Medicaid or CHIP services must develop and enforce network adequacy
standards (42 CFR 438.68(a)).
○ States must at a minimum set quantitative network adequacy standards for the following provider types, if
covered under the managed care contract: primary care, adult and pediatric; obstetrics/gynecology (OB/GYN);
behavioral health (mental health and substance use disorders), adult and pediatric; specialist (as designated by
the state), adult and pediatric; hospital; pharmacy; and pediatric dental (42 CFR 438.68(b)(1)).
○ States that contract with MCPs to provide long-term services and supports (LTSS) must develop a quantitative
network adequacy standard for LTSS provider types (42 CFR 438.68(b)(2)).
• States may elect to use a variety of quantitative standards including, but not limited to, minimum provider-toenrollee ratios; maximum travel time or distance to providers; a minimum percentage of contracted providers that
are accepting new patients; maximum wait times for an appointment; hours of operation (for example, extended
evening or weekend hours); and/or combinations of these quantitative measures (85 Fed. Reg. 72805 (Nov. 13,
2020)).
• In setting network adequacy standards, states must consider the diverse health care needs of their Medicaid and
CHIP populations, including beneficiaries with special needs, individuals living with disabilities, people with limited
English proficiency, and the characteristics of their providers (in terms of training, experience, and specialization)
(42 CFR 438.68(c)).
• States must set network adequacy standards for all geographic areas covered by their contracts with MCPs. States
may set network adequacy standards for the same provider type that vary based on geographic area (42 CFR
438.68(b)(3)).
• States must consider the use of telehealth if needed to ensure sufficient network adequacy when developing their
network adequacy standards (42 CFR 438.68(c)(1)(ix)).
• States must require MCPs that enroll AI/AN beneficiaries to demonstrate that their networks include sufficient
Indian Health Care Providers (IHCPs) so as to ensure timely access to services for AI/AN populations (42 CFR
438.14(b)).
Source: 2020 Medicaid and CHIP managed care final rule, available at
https://www.federalregister.gov/documents/2020/11/13/2020-24758/medicaid-program-medicaid-and-childrens-healthinsurance-program-chip-managed-care.
Note: Medicaid network adequacy standards under 42 CFR 438.68 are adopted for CHIP at 42 CFR 457.1218.
Medicaid network and coverage requirements under 42 CFR 438.14 for AI/AN beneficiaries are adopted for CHIP at
42 CFR 457.1209.

Provider-specific network adequacy standards were added to the Medicaid and CHIP regulations in November 2020
rulemaking. More information about this rulemaking is available at

https://www.federalregister.gov/documents/2020/11/13/2020-24758/medicaid-program-medicaid-and-childrens-health-insuranceprogram-chip-managed-care.

208 | PROTOCOL FOUR

State-defined network adequacy standards must be included in the state’s quality strategy
required under 42 CFR 340(b)(1). States must publish their quality strategies after opportunity
for public comment consistent with transparency requirements in 42 CFR 438.340(c)(1) and (d).
States may work with MCPs to drive improvement in network adequacy and beneficiary access
to care, according to their state quality strategy goals and objectives and quality assessment and
performance improvement (QAPI) programs.
MCPs that operate in multiple states must adhere to each state’s set of quantitative network
adequacy standards. States should annually assess whether their standards continue to align with
the needs of their beneficiary populations, and update their standards as needed.
Medicaid and CHIP MCPs must conduct various activities to assess the adequacy of their
networks as well as maintain provider and beneficiary data sets that allow monitoring of their
networks’ adequacy. States have flexibility in determining the strategies used to assess network
adequacy. Examples include:
•

Geomapping to determine if provider networks meet quantitative standards, such as time and
distance standards

•

Calculation of provider-to-enrollee ratios, by type of provider and geographic area

•

Analysis of in-network and out-of-network utilization data to determine gaps in realized
access (actual use of care)

•

Appointment availability and accessibility studies, such as studies assessing the proportion of
in-network providers accepting new patients or the average wait time for an appointment

•

Telephone surveys or site visits to validate provider directory information

MCPs must share the data, analyses, and results from their network adequacy assessment
activities with EQROs. The activities described below will guide the EQRO in (1) validating the
data and methods used by MCPs to assess network adequacy; (2) validating the results and
generating a validation rating; and (3) reporting the validation findings in the annual EQR
technical report.
This protocol is organized into three phases, which includes the following validation process:
(1) planning, (2) analysis, and (3) reporting. In recognition of changing practices related to the
COVID-19 pandemic, each phase is designed to support activities that may be conducted on-site
or remotely.
While the protocol is written as if the EQRO is validating network adequacy analyses conducted
by the MCP, this protocol also applies to states that opt to conduct the network adequacy
analysis using data submitted by the MCPs or using other data sources. Note that when states
conduct network adequacy analyses under this protocol, the state (1) will conduct an analysis for
each MCP rather than aggregated to the state level and (2) will need to share its analysis as well
as the data used with the EQRO for validation. The EQRO will validate the indicators produced
PROTOCOL FOUR | 209

by the state as if they were calculated by the MCP. In addition, consistent with validation of
other EQR activities, the EQRO may contract with a third-party vendor to conduct the data
validation reported in the annual EQR technical report.
Other protocols, such as Protocol 2 (Validation of Performance Measures), Protocol 3 (Review
of Compliance with Medicaid and CHIP Managed Care Regulations, including availability of
services), and Protocol 5 (Validation of Encounter Data Reported by the Medicaid and CHIP
Managed Care Plan), may provide important information for network adequacy validation. For
example, the EQRO’s compliance findings around availability of services in in Protocol 3 could
inform the validation of network adequacy.

Getting Started on Protocol 4
To complete this protocol, the EQRO undertakes six activities for validating network adequacy
for each MCP (Figure 4.1). These six activities fall into three phases: (1) planning, which
includes defining the scope of the validation (Activity 1), and identifying data sources (Activity
2); (2) analysis, which includes reviewing information systems (Activity 3), and validating data,
methods and results (Activity 4); and (3) reporting, which includes communicating preliminary
findings to MCPs (Activity 5), and submitting findings to the state (Activity 6).

210 | PROTOCOL FOUR

Figure 4.1. Protocol 4 Activities

The steps associated with Activities 1 through 6 are outlined below. Two supplemental resources
are available to help EQROs validate network adequacy:
•

Worksheets for Protocol 4 that EQROs can use to identify network adequacy indicators and
needed data sources for validation; note concerns about MCP data; guide the validation of
MCP network adequacy data, data analysis, and results; and organize findings. States and
PROTOCOL FOUR | 211

EQROs are encouraged, but not required, to use the worksheets. The worksheets are intended
to help structure the way information is collected and reported by EQROs and to promote
compliance with the Network Adequacy Validation protocol.
•

Appendix A. Information System Capabilities Assessment, which can be used to assess the
MCP’s data collection, processing, and reporting systems.

The following toolkits provide additional information and resources for states and EQROs:
•

“Promoting Access in Medicaid and CHIP Managed Care: A Toolkit for Ensuring Provider
Network Adequacy and Service Availability,” which contains additional information and
resources about the validation process. This toolkit is available at
https://www.medicaid.gov/medicaid/downloads/adequacy-and-access-toolkit.pdf.

•

“Promoting Access in Medicaid and CHIP Managed Care: Behavioral Health Provider
Network Adequacy Toolkit,” which contains information focused on network adequacy and
service availability standards for behavioral health. This toolkit is available at
https://www.medicaid.gov/medicaid/downloads/behavior-health-provider-network-adequacytoolkit.pdf.

Activity 1: Define the Scope of the Validation of Quantitative
Network Adequacy Standards
Activity 1 is intended to ensure the EQRO has a complete understanding of the network
adequacy standards and methods used by the state and MCPs to monitor network adequacy. This
activity guides the state and EQRO in outlining all activities that will be conducted as part of the
network adequacy validation.

212 | PROTOCOL FOUR

Step 1: Obtain Needed Information from the
State
In Activity 1, Step 1, the state identifies applicable network
adequacy standards, including (1) quantitative standards
developed by the state under 42 CFR 438.68(b) (adopted by
CHIP at 42 CFR 457.1218); and (2) the standard for MCPs to
include sufficient IHCPs in their provider networks, if the
state enrolls AI/AN beneficiaries in managed care.
Worksheet 4.1 may be used for this purpose.

WORKSHEET 4.1
Resource for Activity 1, Step 1
Worksheet 4.1. State Network Adequacy
Standards to be Validated
• Provides a template to identify all
state network adequacy standards
that must be addressed in the
validation

At the outset of Activity 1, the state must provide the EQRO with the following information, for
validation:
•

Detailed list of the state’s quantitative network adequacy standards, by provider and
plan type, as specified in the state’s contracts with MCPs. This should include network
adequacy standards for adult and pediatric primary care providers, OB/GYNs, adult and
pediatric behavioral health providers, adult and pediatric specialists, hospitals, pharmacists,
pediatric dental providers, and LTSS providers, as applicable. The state should note how
standards vary by MCP type, and across urban, rural, and frontier regions, and provide the
criteria used to designate urban, rural, and frontier regions.

•

Description of network adequacy data and documentation that MCPs submit to the state
to demonstrate compliance with network adequacy standards, including the following:

A list of data and documentation that MCPs submit to the state to demonstrate
compliance

Frequency with which MCPs must submit each type of data and documentation

Any formatting requirements for data and documentation, including file formats for data
files

Any state standards for data completeness and accuracy

Data dictionaries and companion guides, as applicable

•

Description of the information flow from MCPs to the state, including the role of any
contractors or data intermediaries

•

EQR network adequacy validation reports from previous years, if applicable, which can
provide useful data points for determining prior approaches for monitoring and validating
network adequacy, as well as suggest areas for network adequacy improvement or challenges
over time

•

Information about beneficiaries who are dually eligible for Medicare and Medicaid (known
as dually eligible beneficiaries) and covered by a MCP

•

Any other information relevant to network adequacy monitoring and validation
PROTOCOL FOUR | 213

Step 2: Identify and Define Network Adequacy Indicators for Validation
After identifying the quantitative standards that the EQRO
WORKSHEET 4.2
will include in the network adequacy validation, the state
should identify the indicators used by MCPs to assess the
Resource for Activity 1, Step 2
state’s network adequacy standards. Network adequacy
Worksheet 4.2. Network Adequacy
indicators are metrics used to measure plan compliance
Indicators to be Validated
with state’s network adequacy standards (see Box 4.2). The
• Provides a template to identify and
state may establish network adequacy indicators that vary
define the network adequacy
by MCP, or are the same for all MCPs. If the state provides
indicator(s) associated with each
MCPs with guidance around constructing indicators, the
network adequacy standard
state needs to establish clear definitions for each network
adequacy indicator, including criteria for calculating the numerator and denominator. The state
should share this guidance with the EQRO. Worksheet 4.2 can be used for this purpose.
The state and the EQRO will need to address methodological issues that impact indicator
calculations. For example, if the state sets time and distance standards for MCPs, the following
questions to define the indicator for this standard will need to be answered:
•
•

Will distance be measured “as the crow flies” or using
driving distances?
Do network adequacy standards address whether
providers are accessible by public transportation?

•

Are travel times measured separately for private
vehicles, public transit, or other means of transportation?

•

How should travel time calculations address fluctuations
throughout the day or seasonally?

•

How, if at all, is the availability of telehealth services
considered in determining whether an MCP meets the
time and distance network adequacy requirements?
What limits, if any, will be placed on the types and/or
quantity of telehealth services that can contribute toward
network adequacy requirements?

Worksheet 4.2 provides a template for the state and EQRO
to identify and define the network adequacy indicator(s)
associated with each network adequacy standard identified
by the state in Step 1.

214 | PROTOCOL FOUR

Box 4.2. What is the difference
between network adequacy
standards and indicators?
Network adequacy standards are
quantitative parameters that states
establish to set expectations for
contracted managed care plans’
provider networks. For example, a state
may set a network adequacy standard
that all enrollees have access to a
primary care provider (PCP) within 30
miles or 30 minutes of their home.
Network adequacy indicators are
metrics used to measure adherence to
network adequacy standards and to
determine plan compliance with state
network adequacy standards. For the
example given above, the network
adequacy indicator may be the
proportion of enrollees who have access
to a PCP within 30 miles or 30 minutes
of their home.

Step 3: Identify and Define Provider Types
The state may work with the EQRO to identify and define all provider types covered by each
quantitative network adequacy standard to be validated under this protocol. The state and EQRO
may refer to Worksheet 4.1 to see all provider types associated with each network adequacy
standard. If covered under the state’s managed care contracts, the validation should include adult
and pediatric primary care, OB/GYN, adult and pediatric behavioral health, adult and pediatric
specialist, hospital, pharmacy, pediatric dental, and LTSS providers. The state and the EQRO
should also identify additional provider types (e.g., medication-assisted treatment providers for
opioid use disorder), or specialists, as defined by the state, to which the state’s network adequacy
standards apply. The state should provide a method or criteria for categorizing provider types,
and should provide a definition for each provider type. In defining provider types, the state and
the EQRO should address the following questions:
•

Which provider types are authorized by the state to serve as primary care providers (for
example: general practitioner, family physician, internal medicine physician, OB/GYN,
pediatrician, nurse practitioner, physician assistant, or other licensed practitioner as
authorized by the state Medicaid and CHIP program)?

•

Which specialist types (adult and pediatric) are included in the network adequacy standards?

•

How does the state categorize subspecialists? For example, a radiation oncologist could be
categorized as a radiologist or an oncologist.

•

Are there circumstances in which providers who primarily see adult patients count toward
network adequacy standards for the pediatric population? For example, the state may allow a
general psychiatrist to contribute toward network adequacy standards for the pediatric
population, rather than allow only child psychiatrists to contribute toward this requirement.

•

Do the provider types defined in the standards align with the provider types found in the data
that the EQRO will use during the validation process? For example, the EQRO should be
able to crosswalk between the state-defined provider types and the provider types found in
MCP encounter data.

Step 4: Establish Network Adequacy Validation Activities and Timeline
In Step 4, the state and the EQRO should establish the approach for the network adequacy
validation. In this step, the EQRO establishes activities to validate the data, monitoring methods,
results, and reporting from existing MCP network adequacy assessment activities. Depending on
the state’s network adequacy standards and the MCP’s network adequacy assessment activities,
the EQRO may conduct studies to validate the MCP’s results. For example, access and
availability studies may be appropriate to validate information in the MCP’s provider directory
or to assess appointment availability. The state and the EQRO should outline the activities that
the EQRO will complete for the network adequacy validation, and establish a timeline for
completing the network adequacy validation.

PROTOCOL FOUR | 215

Activity 2: Identify Data Sources for Validation
In Activity 2, the EQRO will identify all data sources needed for network adequacy validation,
based on the scope of the validation determined in Activity 1.

Step 1: Identify Data Sources
The EQRO will identify data needed for the validation
activity (see Box 4.3). Typically, data related to both
beneficiaries and providers will be needed. This could
include:
•

Data and documentation from MCPs, such as provider
network data files or directories, beneficiary
enrollment data files, claims and encounter data files,
grievance and appeals data, member experience
survey results, or provider and member handbooks

WORKSHEET 4.3
Resource for Activity 2, Step 1
Worksheet 4.3. Data Sources for
Network Adequacy Validation
• Provides a template to identify all
data sources needed for the
validation

•

Data and documentation from the state, such as board certification status

•

Primary data collection to validate provider directory information or assess appointment
availability and hours of operation
Box 4.3. Examples of Potential Data Sources for Network Adequacy Validation
• Beneficiary enrollment files
• Provider network data files or online provider
directories
• Claims and encounter data
• Case management, authorization, or electronic visit
verification (EVV) systems (particularly for LTSS
providers)
• State board certification registries
• Grievance and appeals data

• Survey data, including member experience surveys,
such as the Consumer Assessment of Health Plan
Survey (CAHPS), Home and Community-Based
Services (HCBS) CAHPS, National Core Indicators,
and National Core Indicators-Aging and Disabilities
• Information from MCP websites, such as provider and
member handbooks
• Primary data collected by the EQRO by telephone,
mail, or in-person visit, such as provider information or
appointment availability

Worksheet 4.3 provides a template that EQROs can use to identify all of the data sources needed
for each network adequacy indicator identified by the state in Activity 1, Step 2.

Step 2: Answer Questions about Data Sources
Consulting with the state as necessary, the EQRO will answer questions about each data source,
such as:
•

Which variables are necessary for network adequacy validation?

216 | PROTOCOL FOUR

Necessary variables in provider datasets may include provider specialty and
subspecialty, office address(es), languages spoken, disability access, and hours of
service

Necessary variables in beneficiary datasets may include home address, eligibility
category, date of birth, sex, primary language spoken, race, ethnicity, disability status,
and enrollment/termination dates

•

How are the files and data formatted? Will the EQRO need to convert the files and data to
other formats and, if so, what potential challenges exist and why?

•

Does the state set standards for completeness and accuracy of the data?

•

Should the EQRO expect to receive data dictionaries or other supplemental resources with
the data?

•

What challenges could the EQRO encounter in accessing and using the data and how does
the challenges impact the data assessment?

Worksheet 4.3 provides a template that EQROs can use to record the pertinent information
related to the data sources needed.

Activity 3: Review Information Systems Underlying Network
Adequacy Monitoring
In Activity 3, the EQRO will determine if the MCP’s information systems are able to collect and
report accurate data related to each network adequacy indicator. The EQRO will assess the
information system in three steps:
1. Review the MCP’s most recently completed Information System Capacity Assessment
(ISCA)
2. Assess processes for collecting network adequacy validation data not addressed in the ISCA
3. Interview MCP or other personnel to clarify findings

Step 1: Review the MCP’s ISCA
The EQRO should determine whether the MCP has completed
an ISCA review within the previous two years. If the MCP has
completed an ISCA within the previous two years, the EQRO
should review the findings. If the MCP has not conducted an
ISCA within the previous two years, the EQRO must conduct
one consistent with the processes discussed in Appendix A.
Based on its review of the statement of findings from the ISCA,
the EQRO should understand the IT system architecture, file
structure, information flow, data processing procedures, and

WORKSHEET 4.4
Resource for Activity 3, Step 1
Worksheet 4.4. Network Adequacy Data
Concerns Identified in Review of ISCA
• Provides a template to describe data
concerns identified in the EQRO’s
review of the MCP’s Information
System Capacity Assessment (ISCA)

PROTOCOL FOUR | 217

completeness and accuracy of data related to current provider networks. The EQRO should
review the ISCA to understand processes for maintaining and updating provider data, including
how the MCP tracks providers over time, across multiple office locations, and through changes
in participation in the MCP’s network; how the MCP identifies providers or organizations
excluded from the Medicaid and CHIP program each month (e.g., List of Excluded Individuals
and Entities); and whether the MCP requires its provider network to update provider data each
month. If the EQRO identifies issues that may contribute to inaccurate or incomplete network
adequacy data, the EQRO should list any concerns about the data in Worksheet 4.4. The
statement of findings from the ISCA includes an assessment of the utility of the MCP’s
information system for review of provider network adequacy. The EQRO should review this
assessment and the ISCA in full to identify weaknesses in the MCP’s information systems
required for network adequacy validation, including where and how information systems may be
vulnerable to incomplete or inaccurate data capture or processing, integration, storage, or
reporting.

Step 2: Assess Processes for Collecting Network Adequacy Data Not
Addressed in the ISCA
The EQRO should identify any data sources included in the
network adequacy validation that were not reviewed in the ISCA.
This may include primary data the EQRO will collect for the
network adequacy validation (such as access and availability
study data) and data from non-MCP entities (such as the state or
third-party vendors), as well as any data sources from the MCP
not covered in the ISCA. The EQRO should assess the integrity
of the systems that collect, store, and process this data. For each
data source not reviewed in the ISCA, the EQRO should consider
factors such as:

WORKSHEET 4.5
Resource for Activity 3, Step 2
Worksheet 4.5. Assessment of network
adequacy data sources not reviewed in
the ISCA
• Provides a template to assess the
integrity of the MCP’s system(s) that
collects, stores, and processes
network adequacy data not
addressed in the ISCA

•

What system is used to collect and store this data?

•

How are data entered (manually or electronically)?

•

Which staff are involved in collecting, storing, and analyzing this data, and what is their level
of training? 64

•

Are there adequate staffing resources to collect and analyze data?

•

How frequently are the data collected and updated?

Level of training refers to both formal and informal training that qualifies staff to collect, store, and analyze network adequacy
data sources.

218 | PROTOCOL FOUR

•

What errors may occur in the process of collecting, storing, and analyzing the data? What
quality assurance systems are in place to prevent and fix these errors?

•

To what extent are data missing?

Worksheet 4.5 can be used to record any issues identified by the EQRO that may contribute to
inaccurate or incomplete network adequacy data.

Step 3: Interview MCP or Other Personnel
After assessing the data sources needed for the network adequacy validation, the EQRO should
conduct follow-up interviews with MCP personnel, as needed, to supplement its understanding
of the MCP’s information systems and processes. As necessary, the EQRO should follow up
with personnel in other agencies or organizations with any questions about information systems
and processes used in the network adequacy validation that are not maintained by the MCP.

Activity 4: Validate Network Adequacy Assessment Data,
Methods, and Results
In Activity 4, the EQRO evaluates each MCP’s ability to
(1) collect reliable and valid network adequacy
monitoring data, (2) use sound methods to assess the
adequacy of its managed care networks, and (3) produce
accurate results to support MCP and state network
adequacy monitoring efforts. The EQRO also generates
a validation rating for each network adequacy indicator
for each MCP. The approach taken by the EQRO will
vary based on the network adequacy methods used by
MCPs, the network adequacy standards and indicators
specified by the state, and the network adequacy
validation scope defined by the state (Activity 1).

WORKSHEET 4.6
Resource for Activity 4, Steps 1-3
Worksheet 4.6. Assessment of MCP
Network Adequacy Data, Methods, and
Results
• Provides a template for reviewing the
data, methods and results, and
generating a validation rating for each
network adequacy indicator

For Activity 4, Steps 1-3, Worksheet 4.6 providers a template that EQROs can use to assess the
MCP’s data, methods, and results for each network adequacy indicator. Worksheet 4.6 also
guides the EQRO to generate a validation rating for each network adequacy indicator.

Step 1: Assess the Reliability and Validity of MCP Network Adequacy Data
The EQRO will determine if the data used by MCPs to monitor network adequacy are accurate
and current so as to generate meaningful, actionable results. Worksheet 4.6 guides the EQRO in
assessing MCP network adequacy data for each indicator. The EQRO should:

PROTOCOL FOUR | 219

•

Determine whether the required variables identified in Activity 2 are included in each data
source. For each variable, the EQRO should calculate the proportion of missing data, and
note any patterns in missing data

•

Review the MCP’s process for updating data (for example, updating provider and beneficiary
information) and the frequency with which data are updated

•

Assess changes in the MCP’s data systems that might affect the accuracy or completeness of
network adequacy monitoring data (e.g., major upgrades, consolidations within the system,
acquisitions/mergers with other MCPs)

The EQRO should also compare the MCP’s data from previous years against the most recent
data to assess reliability, as major changes over time could reflect data quality issues. If
significant changes are noted, the EQRO should consult with the MCP to determine if external
factors (such as changes in populations covered or demographic changes in the overall state
population) may be the cause.
If an MCP uses performance measures or encounter data in its network adequacy monitoring
activities, the EQRO can use the results of Protocol 2 (Validation of Performance Measures)
and/or Protocol 5 (Validation of Encounter Data Reported by the Medicaid and CHIP Managed
Care Plan) to inform the EQRO’s assessment of data reliability and validity.

Step 2: Assess the Methods Used by the MCP to Assess Network Adequacy
The EQRO should next review the methods used by the MCP to calculate each network
adequacy indicator. For each network adequacy indicator, the EQRO should consider questions
such as:
•

Are the network adequacy monitoring methods selected by the MCP to calculate this
indicator appropriate to the state Medicaid and CHIP population(s)? Are these methods likely
to generate the data needed to calculate this indicator?

•

If the MCP is sampling a subset of the Medicaid and/or CHIP provider or beneficiary
population to calculate this indicator, was the sample drawn using statistically valid sampling
techniques? Is the sample representative of the population? Are sample sizes for the
validation sufficient to draw statistically significant conclusions?

•

In calculating this indicator, does the MCP use a system for classifying provider types that
follows how the state designates provider types as specialists in the MCP’s contract?

•

Does the MCP consider the availability of telehealth services in its assessment of network
adequacy? Does the MCP’s approach for addressing telehealth match the state’s
expectations?

•

Are the methods used by the MCP to calculate this indicator rigorous and objective or could
the methods be subject to manipulation?

220 | PROTOCOL FOUR

Additionally, the EQRO may need to consider questions that apply to specific types of indicators
and/or methods the MCP may use. As applicable, the EQRO should consider questions such as
the following:
•

Considerations for time and distance analyses:

How is distance measured (“as the crow flies” or using road distances)?

How is time measured (e.g., during low traffic or high traffic time periods, using driving
distance or public transit)?

Does the MCP’s approach match the state’s expectations (provided in Activity 1, Step
2)?

•

Does the MCP’s approach for deriving provider-to-enrollee ratios and/or minimum
percentage of contracted providers accepting new patients match the state’s expectation?

•

Does the MCP’s approach for determining the maximum wait time for an appointment match
the state’s expectation?

Worksheet 4.6 guides the EQRO in evaluating the data and methods used by MCPs to assess the
results generated by each MCP for each indicator.

Step 3: Validate Network Adequacy Results Submitted by the MCP
In Step 3, the EQRO should validate the calculations and results generated by the MCP for each
network adequacy indicator. The EQRO should assess whether the results are valid, accurate,
and reliable, and if the MCP’s interpretation of data was accurate. The EQRO’s approach for
validating network adequacy indicators should follow the scope outlined in Activity 1. Often, the
EQRO will use the data sources provided by the MCP to reproduce the MCP’s calculations. As
applicable, the EQRO may also conduct studies, such as provider directory validation studies and
appointment availability studies, to validate the MCP’s results.
Worksheet 4.6 can be used to guide the EQRO in assessing the MCP’s network adequacy results
and generating validation ratings that reflect the EQRO’s overall confidence that the MCO used
an acceptable methodology in the design, data collection, analysis and interpretation of each
network adequacy indicator.

PROTOCOL FOUR | 221

Step 4: Summarize Network Adequacy Validation Findings
Next, the EQRO will review the indicators validated in
Activity 4, Steps 1-3 against the list of state network
adequacy standards and indicators established in Activity 1.
The EQRO will indicate whether the MCP addresses all
state network adequacy indicators in its network adequacy
monitoring activities, provide the validation rating for each
indicator, and note any network adequacy indicators that
could not be validated due to missing or incomplete data.
The EQRO can use Worksheet 4.7 to summarize its
findings.

WORKSHEET 4.7
Resource for Activity 4, Step 4
Worksheet 4.7. Summary of network
adequacy validation findings
• Provides a template for summarizing
all validation findings, including
validation ratings and comments for
network adequacy indicator

Activity 5: Communicate Preliminary Findings to Each MCP
In Activity 5, the EQRO will share preliminary network adequacy validation findings with each
MCP, and correct omissions and errors if necessary. The EQRO should prepare a preliminary
validation report for each MCP. In the report, the EQRO should document findings, provide
validation ratings, identify areas of concern, and make suggestions for improvement. The EQRO
should submit its preliminary findings to the MCP. The MCP may provide documentation to
correct errors and omissions in the preliminary report. As needed, the EQRO may discuss the
documentation with each MCP.

Activity 6: Submit Findings to State
After completing Activities 1 through 5, the EQRO should compile the results for each MCP into
the annual EQR Technical Report. In the report, the EQRO will provide its assessment of each
MCP’s ability to (1) collect reliable and valid network adequacy monitoring data, (2) use sound
methods to assess the adequacy of its managed care networks, and (3) produce accurate results to
support MCP and state network adequacy monitoring efforts. EQROs are encouraged to present
this information in a summary narrative format with supporting tables and graphics, where
appropriate. Please see “Tips for Drafting EQR Reports” in the introduction to the protocols for
further guidance about how to produce a clear and concise report. In addition, please refer to the
worksheets for guidance on the content to include in the report.
The EQRO’s technical report to the state should follow the state’s required format, and include
the following elements, along with worksheets, tools, and other supporting documentation:
•

A description of the state’s network adequacy standards for provider types covered by the
state’s MCPs, including minimum quantitative network adequacy standards, and the network
adequacy indicators that were validated for each MCP (Worksheet 4.1 and Worksheet 4.2)

222 | PROTOCOL FOUR

•

A list of the data and documentation validated by the
EQRO (Worksheet 4.3)

Resources for Activity 6

•

A description of the EQRO’s validation activities
including:

Information gathered in Activities 1
through 4 using the following
worksheets may be helpful when
preparing the final validation report:

The EQRO team members involved in the
validation and other participants (MCP staff,
vendors)

A summary of the validation strategy

Methods for collecting primary data

Other considerations relevant to the network
adequacy validation process

•

Data analysis methodology

Findings on the MCP’s information systems
capabilities and data integration, including
documentation of the timing of the state’s most recent
ISCA and a description of what documentation was
reviewed by the EQRO to support the validation of
network adequacy (Worksheet 4.6)
Analyses and conclusions for each network adequacy
validation activity for each MCP. The EQRO should
compile data across MCPs to create data tables with
summary statistics for each MCP that include actual
results and validation ratings (and analysis of patterns
across MCPs), using the criteria established by the state
in Activity 1 (Worksheet 4.7)
Recommendations for improving the reliability and
validity of each MCP’s process for monitoring network
adequacy, including implications for the MCP’s data
systems, methods, and staffing (e.g., programming and
analytic capacity) (Worksheet 4.8)

Worksheet 4.1. State Network
Adequacy Standards to be Validated
• Lists all state network adequacy
standards that must be addressed in
the validation
Worksheet 4.2. Network Adequacy
Indicators to be Validated
• Lists and defines the network
adequacy indicator(s) associated with
each network adequacy standard
Worksheet 4.3. Data Sources for
Network Adequacy Validation
• Lists all data sources included in the
validation
Worksheet 4.6. Assessment of MCP
Network Adequacy Data, Methods,
and Results
• Provides a template for reviewing the
data, methods and results for each
network adequacy indicator
Worksheet 4.7. Summary of Network
Adequacy Validation Findings
• Provides validation ratings and
comments for network adequacy
indicator
Worksheet 4.8. Recommendations to
Improve MCP Assessment of Network
Adequacy
• Provides a template for summarizing
EQRO recommendations from past
EQR technical reports and EQRO
recommendations based on the
current validation process

When possible, the validation report should identify
recommendations from the previous year’s report submitted to the state, and discuss progress
made on these recommendations over the past year based on information gathered during the
validation process (Worksheet 4.8).
END OF PROTOCOL 4

PROTOCOL FOUR | 223

Worksheets For Protocol 4: Network Adequacy
Validation Tools
Instructions. Use these or similar worksheets to identify the network adequacy indicators to be validated, document
and describe the data sources used for validation, assess the underlying data structures and considerations, validate
network adequacy data elements, and compare the network adequacy findings to state network adequacy standards.
This tool includes the following worksheets cross-walked to the applicable activity and step. Examples are provided
for the first four worksheets for illustrative purposes.
Worksheet name
Worksheet 4.1. State Network Adequacy
Standards to be Validated

Protocol activity and step
Activity 1. Step 2. Obtain Needed Information from the State

Worksheet 4.2. Network Adequacy Indicators to be Activity 1. Step 2. Identify and Define Network Adequacy
Validated
Indicators for Validation
Worksheet 4.3 Data Sources for Network
Adequacy Validation

Activity 2. Step 1. Identify Data Sources

Worksheet 4.4. Network Adequacy Data Concerns
Identified in Review of ISCA

Activity 3. Step 1. Review the MCP’s ISCA

Worksheet 4.5. Assessment of Network Adequacy
Data Sources not Reviewed in the ISCA

Activity 3. Step 2. Assess Processes for Collecting Network
Adequacy Data Not Addressed in the ISCA

Worksheet 4.6. Assessment of MCP Network
Adequacy Data, Methods, and Results

Activity 4. Step 1. Assess the Reliability and Validity of MCP
Network Adequacy Data
Activity 4. Step 2. Assess the Methods Used by the MCP to
Assess Network Adequacy
Activity 4. Step 3. Validate Network Adequacy Results
Submitted by the MCP

Worksheet 4.7. Summary of Network Adequacy
Validation findings

Activity 4. Step 4. Summarize Network Adequacy Validation
Findings

Worksheet 4.8. Recommendations to Improve
MCP Assessment of Network Adequacy

Activity 6. Submit Findings to the State

For each MCP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
Plan type

□ MCO □ PIHP □ PAHP □ LTSS
□ Other: specify __________________________________
If PIHP or PAHP, type: _____________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only)
□ Medicaid and CHIP

224 | PROTOCOL FOUR

Worksheet 4.1. State Network Adequacy Standards to be Validated
Instructions: Worksheet 4.1 guides the state and the EQRO to identify the network adequacy standards that the
EQRO will need to validate. In the table below, the EQRO should list the quantitative network adequacy standards to
be validated under this protocol. If covered under the state’s managed care contracts, the validation standards should
include adult and pediatric primary care, OB/GYN, adult and pediatric behavioral health, adult and pediatric specialist,
hospital, pharmacy, pediatric dental, and LTSS providers. The validation standards should also include additional
provider types (e.g., medication-assisted treatment providers for opioid use disorder), or specialists, as defined by the
state, that follow the state’s network adequacy standards. The state and the EQRO should add rows as necessary to
the table to capture all state network adequacy standards that will be validated. Definitions for this activity include:
•

Network adequacy standard: A quantitative parameter that states establish to set expectations for contracted
managed care plans’ provider networks. For example, a state may set a network adequacy standard that all
enrollees have access to a primary care provider (PCP) within 30 miles or 30 minutes of their home.

•

Applicable provider types: All provider types to which the network adequacy standard applies.

•

Applicable plan types: All plan types (such as Medicaid, CHIP, LTSS, and dental plans) to which the network
adequacy standard applies.

•

Applicable regions: All regions to which the network adequacy standard applies. Typically, regions are
categorized as urban, rural and frontier. In Activity 1, Step 1, the state and EQRO should clarify how regions are
defined. When standards differ by region (for example, if the state’s distance standard between a beneficiary
home and primary care provider is 30 miles in urban areas and 50 miles in rural areas), they should be listed in
separate rows in the table below.

•

Data and documentation submitted by MCPs: All data and documentation MCPs must submit to demonstrate
compliance with the network adequacy standard. In parentheses, please note the frequency with which this data
and documentation is submitted (e.g., annually, quarterly, monthly).

Network adequacy
standard
Enrollees must have
access to a primary care
provider office within 30
minutes or 30 miles of
their residence

Applicable
provider types
Primary care
(family medicine
physicians,
internal medicine
physicians,
OBGYNs,
pediatricians,
nurse
practitioners,
physician
assistants)

Applicable plan
types
Medicaid, CHIP

Applicable
regions
Statewide

Data and documentation
submitted by MCPs
(frequency)
Beneficiary enrollment files
(monthly)
Provider network data files
(quarterly)

PROTOCOL FOUR | 225

Worksheet 4.2. Network Adequacy Indicators to be Validated
Instructions: Worksheet 4.2 guides the state and the EQRO to define the network adequacy indicators that the
EQRO will need to validate. To start, the EQRO should fill in the first column of the table below with the network
adequacy standards identified in Activity 1, Step 2 (Worksheet 4.1). The state and the EQRO should then identify and
define the indicator(s) that will be validated, listing each indicator in its own row and adding rows as necessary. A
separate worksheet should be completed to define the indicators that will be validated for each MCP, taking into
account the standards that apply to each plan type. Definitions for this activity include:
•

•

Network adequacy indicator: The metric(s) used to assess adherence to the quantitative network adequacy
standard required by the state. For example, the network adequacy indicator for a network adequacy standard
that all enrollees have access to a primary care provider (PCP) within 30 miles or 30 minutes of their home could
be the proportion of enrollees who have access to a primary care provider within 30 miles or 30 minutes from their
home.

•

Definition of network adequacy indicator: A clear description of the network adequacy indicator, including
criteria for calculating the numerator and denominator. The definition should address specific methodological
issues that impact indicator calculations. For example, for time and distance indicators, the definition should
specify whether distance is measured “as the crow flies” or using driving distances. The definition should also
identify the provider types to which the indicator applies.

Managed Care Plan (MCP) name: Alpha Plan
Network adequacy
standard
Beneficiaries must have
access to a primary care
provider office within 30
minutes or 30 miles of their
residence

Network adequacy
indicator
Proportion of beneficiaries
who have a primary care
provider accepting new
Medicaid patients within
30 minutes or 30 miles of
their residence

Definition of network adequacy indicator
Numerator: Number of beneficiaries for which one or
more of the following is true:
•

An in-network provider office is a 30-minute drive
or less from their residence (according to mapping
software)

•

An in-network provider office is 30 miles or less by
road from of their home (according to mapping
software)
Denominator: All Medicaid and CHIP beneficiaries
except those enrolled only in LTSS plans

226 | PROTOCOL FOUR

Worksheet 4.3. Data Sources for Network Adequacy Validation
Instructions: For each network adequacy indicator identified in Activity 1, Step 2 (Worksheet 4.2), Worksheet 4.3
lists the network adequacy indicators used to measure the MCPs’ compliance with the network adequacy standards
established by the state and guides the EQRO to identify all data sources needed to validate a network adequacy
indicator. To start, the EQRO should fill in the first column of the table below with the network adequacy indicators
identified in Worksheet 4.2, adding rows as necessary. If multiple data sources will be used to validate a given
indicator, each data source should be listed in a separate row. The EQRO should then fill in the remaining columns
with information about the data source. Definitions for this activity include:
•

Network adequacy indicator: The metric(s) used to assess adherence to the quantitative network adequacy
standard required by the state. For example, the network adequacy indicator may be that enrollees have access
to a primary care provider within 30 miles or 30 minutes from their home. The table below should include all
network adequacy indicators identified in Worksheet 4.2.

•

Data source: The type of data needed to validate a network adequacy indicator. When multiple data sources are
used to validate a given indicator, each data source should be listed in a separate row. For example, if validation
of time and distance indicators requires both provider network and beneficiary enrollment files, each data source
should be listed separately. The year(s) of data should also be listed.

•

Data format and software: File format for the data source and any digital software needed to access or analyze
this file format. Additionally, the EQRO should note if it will need to convert this data to other file formats, and if
so, any potential challenges that may occur.

•

Variables for network adequacy validation: All variables within the data source that are needed to complete
the validation activity. The EQRO should consider how to utilize different variables for beneficiary datasets and
provider datasets.

•

State standards for data accuracy, timeliness, and completion: If applicable, any standards set by the state
related to data accuracy and completeness. Typically, this applies to data that MCPs collect and submit to the
state.

•

Challenges and notes: Any potential challenges the EQRO could encounter in accessing and using the data
source, and any additional information that provides context for data validation of the given indicator. If applicable,
this column could include hyperlink(s) to the data source or related materials to facilitate validation of the given
indicator.

Managed Care Plan (MCP) name: Alpha Plan

Network
adequacy
indicator

Data source
and year(s) of
data

Proportion of
beneficiaries
who have a
primary care
provider office
within 30
minutes or 30
miles of their
residence

Beneficiary
enrollment files

Data format and
analysis software;
note if conversion
required
Comma Separated
Value (CSV)

Variables for
network
adequacy
validation
Beneficiary
address,
beneficiary
date of birth,
beneficiary
plan type

State standards
for accuracy,
timeliness, and
completion
State requires
MCPs to submit
updated and
accurate
beneficiary
enrollment files
monthly

Challenges and
notes
State and MCP
have noted that in
urban regions a
significant
proportion of
beneficiaries rely
on public transit,
rather than driving

PROTOCOL FOUR | 227

Network
adequacy
indicator

Data source
and year(s) of
data

Proportion of
beneficiaries
who have a
primary care
provider office
within 30
minutes or 30
miles of their
residence

Provider
network data
files

Data format and
analysis software;
note if conversion
required
Comma Separated
Value (CSV)

Variables for
network
adequacy
validation
Provider
address,
provider type

State standards
for accuracy,
timeliness, and
completion
State requires
MCPs to submitted
updated and
accurate provider
network data files
quarterly. The state
flags and rejects
data in which
provider type is not
specified.

Challenges and
notes
State and MCP
have noted
challenges
keeping provider
network data upto-date; provider
network data also
does not include
information about
accommodations
for beneficiaries
with physical
disabilities or low
English
proficiency

228 | PROTOCOL FOUR

Worksheet 4.4. Network Adequacy Data Concerns Identified in Review of ISCA
Instructions: Worksheet 4.4 guides the EQRO in identifying any data concerns it has identified in its review of an
MCP’s Information System Capacity Assessment (ISCA). The EQRO should first determine whether the MCP has
completed an ISCA review within the past two years. If the MCP has not conducted an ISCA within the previous two
years, the EQRO must conduct one consistent with the processes discussed in Appendix A. If the MCP has
completed an ISCA review within the past two years, the EQRO should review the findings and identify any concerns
related to data sources that will be used in the network adequacy validation.
The EQRO should fill in the first column of the table below with data sources identified in Activity 2, Step 1
(Worksheet 4.3) that are covered in the ISCA. If the EQRO identifies concerns related to a given data source in its
review of ISCA findings, the EQRO should fill in the remaining columns to describe the concern and potential
workarounds. If no data concerns are identified for a given data source, the EQRO should enter “Not identified” in the
second column.
Managed Care Plan (MCP) name: Alpha Plan

Provider
network data
files

Provider network data
X
files may be inaccurate
due to providers
entering and leaving
networks, or changes in
provider information,
such as address

Other

Data reporting

Data storage

Data integration

Data concern

Data processing

Data source

Data capture

Type (check boxes)

Potential solutions or
workarounds
The EQRO will validate a
sample of providers through
phone calls or on-site visits to
determine if the provider still
participates in the network, if
the location is accurate, and if
the provider is accepting new
Medicaid patients.

PROTOCOL FOUR | 229

Worksheet 4.5. Assessment of Network Adequacy Data Sources not Reviewed in the ISCA
Instructions: Worksheet 4.5 guides the EQRO in assessing the integrity of any systems that collect, store, and
process network adequacy data not addressed in the ISCA. The EQRO should identify any data source(s) identified
in Activity 2, Step 1 (Worksheet 4.3) that were not reviewed in the ISCA. This may include MCP data sources not
covered in the ISCA, data from non-MCP entities, and primary data the EQRO plans to collect for the purpose of the
network adequacy validation. For each data source, the EQRO should complete the table below to assess the
integrity of the system that collects, stores, and processes the data. The EQRO should conduct follow-up interviews
as needed to supplement its understanding of the information systems and processes.
Managed Care Plan (MCP) name: ________________________________
Name of data source
What system is used to collect this data?
What system is used to store this data?
How frequently are the data collected and
updated?
What software systems and/or programming
languages are used to analyze this data?
Which staff are involved in collecting and storing
this data, and what is their level of training?
Are there adequate staffing resources to collect
and analyze data? Specifically, does the MCP
employ enough data analysts and do they have
adequate time to perform necessary analytics?
Which staff are involved in analyzing and
reporting this data, and what is their level of
training?
What errors may occur in the process of
collecting, storing, and analyzing the data?
What systems are in place to prevent and fix
errors that occur in the process of collecting,
storing, and analyzing the data?
What proportion of the data are missing or
incomplete on key data elements?
What systems are in place to prevent missing or
incomplete data?
Data concerns relevant to network adequacy
validation
Potential solutions or workarounds to address
data concerns
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

230 | PROTOCOL FOUR

Worksheet 4.6. Assessment of MCP Network Adequacy Data, Methods, and Results
Instructions: Worksheet 4.6 guides the EQRO in evaluating and assessing the data and methods used by MCPs to
calculate results generated for each network adequacy indicator. This worksheet also guides the EQRO in generating
a validation rating that reflects the EQRO’s overall confidence that an acceptable methodology was used for all
phases of design, data collection, analysis and interpretation of the network adequacy indicator.
The EQRO should fill in the table below for each network adequacy indicator to be validated. The EQRO should
respond to the questions below, and insert comments to explain “No” and “Not Applicable” responses. If an item is
partially met, select “No” and explain in comments. For example, if data sources are available for some but not all
indicators or for some but not all years, select “No” and explain in comments. If an item is “Not Applicable,” please
explain in comments.
Managed Care Plan (MCP) name: ________________________________
Network Adequacy Indicator: ___________________________________
Question

Yes

Not Applicable

Comments

Assessment of data collection
procedures
Were all data sources (and year[s] of
data) needed to calculate this indicator
submitted by the MCP to the EQRO?
For each data source, were all variables
needed to calculate this indicator
included?
Are there any patterns in missing data
that may affect the calculation of this
indicator? (Note: This assessment
should be based on a systematic
assessment of the proportion of missing
data for each variable.)
Do the MCP’s data enable valid, reliable,
and timely calculations of this indicator?
Did the MCP’s data collection
instruments and systems allow for
consistent and accurate data collection
for this indicator over the time periods
studied?
During the time period included in the
reporting cycle, have there been any
changes in the MCP’s data systems that
might affect the accuracy or
completeness of network adequacy data
used to calculate this indicator (e.g.,
major upgrades, consolidations within
the system, acquisitions/mergers with
other MCPs)?

231 | PROTOCOL FOUR

Question
If encounter or utilization data were used
to calculate this indicator, did providers
submit data for all encounters?
If LTSS data were used to calculate this
indicator, were all relevant LTSS
provider services included (for example,
through claims and encounter data,
authorization systems, case
management systems, or electronic visit
verification [EVV] systems)?
If access and availability studies were
conducted to calculate this indicator,
does the MCP include all phone calls
made in the denominator? This means
phone calls that do not reach a provider
office may be excluded from the
denominator.
If access and availability studies were
conducted to calculate this indicator,
does the MCP have processes for
addressing potential roadblocks in
identification, such as lack of a Medicaid
or CHIP ID or medical record number
needed to speak with provider offices?
Assessment of MCP Network
Adequacy Methods
Are the methods selected by the MCP to
calculate this indicator appropriate for
the state
Are the methods selected by the MCP to
calculate this indicator appropriate to the
state Medicaid and CHIP population(s)?
Are the methods selected by the MCP
adequate to generate the data needed to
calculate this indicator?
In calculating this indicator, does the
MCP use a system for classifying
provider types that matches the state’s
expectations and follows how the state
defines a specialist?
If applicable, does the MCP’s approach
for addressing telehealth match the
state’s expectations?

232 | PROTOCOL FOUR

Yes

Not Applicable

Comments

Question

Yes

Not Applicable

Comments

If the MCP is sampling a subset of the
Medicaid and/or CHIP population to
calculate this indicator, did the sampling
frame contain a complete, recent, and
accurate list of the target population?
A sampling frame is the list from which
the sample is drawn. It includes the
universe of members of the target
population, typically Medicaid and CHIP
beneficiaries and providers. The
completeness, currency, and accuracy
of the sampling frame are key to the
representativeness of the sample.
If the MCP is sampling a subset of the
Medicaid and/or CHIP population to
calculate this indicator, is the sample
representative of the population?
If the MCP is sampling a subset of the
Medicaid and/or CHIP population to
calculate this indicator, are sample sizes
large enough to draw statistically
significant conclusions?
In calculating this indicator, were valid
sampling techniques used to protect
against bias? Specify the type of
sampling used in the “comments” field.
If applicable to this indicator, does the
MCP’s approach for measuring distance
(e.g., “as the crow flies” or using road
distances) match the state’s
expectation?
If applicable to this indicator, does the
MCP’s approach for measuring time
(e.g., during low traffic or high traffic time
periods, using driving distance or public
transit) match the state’s expectation?
If applicable to this indicator, does the
MCP’s approach to deriving provider-toenrollee ratios or percentage of
contracted providers accepting new
patients match the state’s expectation?
If applicable to this indicator, does the
MCP’s approach for determining the
maximum wait time for an appointment
match the state’s expectation?
Are the methods used to calculate this
indicator rigorous and objective?

PROTOCOL FOUR | 233

Question

Yes

Not Applicable

Comments

Are the methods used to calculate this
indicator unlikely to be subject to
manipulation? If “no,” please describe in
the “comments” field.
Assessment of MCP network
adequacy results
In calculating this indicator, did the MCP
produce valid results—that is, did the
MCP measure what they intended to
measure?
In calculating this indicator, did the MCP
produce accurate results—that is, did
the MCP’s calculated values reflect the
true values?
In calculating this indicator, did the MCP
produce reliable results—that is, were
the MCP’s results reproducible and
consistent?
In calculating this indicator, did the MCP
accurately interpret its results?
Comments
Please note any recommendations for
improving the data collection procedures
to calculate this indicator.
Please note any recommendations for
improving the sampling methods to
calculate this indicator.
Please note any recommendations for
improving the analysis to calculate this
indicator.
Please note any recommendations for
improving the results to calculate this
indicator.

Calculate validation score:
A. Total number of “Yes” responses
B. Total number of “No” responses
Score = A / (A + B) x 100

Determine validation rating:
The “validation rating” refers to the EQRO’s overall confidence that acceptable methodology was used for all phases
of design, data collection, analysis, and interpretation of the network adequacy indicator.

234 | PROTOCOL FOUR

Validation score

Validation rating

90.0% or greater

High confidence

51.0% to 89.9%

Moderate confidence

10.0% to 49.9%

Low confidence

Less than 10%

No confidence

Summary:
Managed Care Plan (MCP) name:
Indicator:
Validation rating:





High confidence
Moderate confidence
Low confidence
No confidence

Comments
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL FOUR | 235

Worksheet 4.7. Summary of Network Adequacy Validation Findings
Instructions: Worksheet 4.7 guides the EQRO in summarizing its validation findings. The EQRO should complete
this worksheet separately for each MCP. To start, the EQRO should fill in the first column of the table below with the
network adequacy indicators identified in Activity 1, Step 2 (Worksheet 4.2). The EQRO should then note whether the
MCP addressed the network adequacy indicator in its network adequacy assessment activities. For indicators
addressed by the MCP, the EQRO should provide the validation rating generated in Activity 4, Step 3 (Worksheet
4.6), noting if any indicators could not be validated due to missing data or other issues. The EQRO may provide any
additional context needed in the “comments” field. The EQRO should add additional rows as needed to include all
network adequacy indicators. Definitions for this worksheet include:
•

Network adequacy indicator: The metric(s) used to assess adherence to the quantitative network adequacy
standard required by the state. For example, the network adequacy indicator may be the proportion of enrollees
who have access to a primary care provider within 30 miles or 30 minutes from their home, or provider-to-enrollee
ratio. The table below should include all network adequacy indicators identified in Activity 1, Step 2 (Worksheet
4.2).

•

Validation rating: The rating, calculated in Activity 4, Step 3 (Worksheet 4.6) that refers to the EQRO’s overall
confidence that acceptable methodology was used for all phases of design, data collection, data analysis, and
interpretation of network adequacy monitoring activities.

Managed Care Plan (MCP) name: ________________________________

Network
adequacy
indicator

Did the MCP address
this indicator in its
network adequacy
monitoring activities?

Validation rating

 Addressed
 Missing







High confidence
Moderate confidence
Low confidence
No confidence
Could not be validated

 Addressed
 Missing







High confidence
Moderate confidence
Low confidence
No confidence
Could not be validated

 Addressed
 Missing







High confidence
Moderate confidence
Low confidence
No confidence
Could not be validated

 Addressed
 Missing







High confidence
Moderate confidence
Low confidence
No confidence
Could not be validated

 Addressed
 Missing

 High confidence
 Moderate confidence
 Low confidence

Comments

236

Network
adequacy
indicator

Did the MCP address
this indicator in its
network adequacy
monitoring activities?

Validation rating

Comments

 No confidence
 Could not be validated
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL FOUR | 237

Worksheet 4.8. Recommendations to Improve MCP Assessment of Network Adequacy
Instructions: Worksheet 4.8 provides a template for the EQRO to refer back to EQRO recommendations from past
EQR technical reports (where applicable), review MCP progress in responding to those recommendations, and
provide recommendations based on the current network adequacy validation cycle. The recommendations should be
specific and actionable to support improvement of the MCP’s assessment of network adequacy.
Managed Care Plan (MCP) name:
Prior Recommendation Year (if applicable):
EQRO Prior Recommendations (if applicable):

Summary of MCP Response to Prior Recommendations (if applicable):

EQRO Assessment of Degree to which MCP Effectively Addressed the Recommendations (if applicable):

Current Recommendation Year:
EQRO Current Recommendations for MCP Assessment of Network Adequacy:

END OF WORKSHEETS FOR PROTOCOL 4

238 | PROTOCOL FOUR

Protocol 5. Validation of Encounter Data
Reported by the Medicaid and CHIP
Managed Care Plan
An optional EQR-related activity

ACTIVITY 2: REVIEW THE MCP’S CAPABILITY
ACTIVITY 3: ANALYZE ELECTRONIC ENCOUNTER DATA
ACTIVITY 4: REVIEW MEDICAL RECORDS
ACTIVITY 5: SUBMIT FINDINGS

Background
Encounter data are the information related to the receipt of any item or
service by an enrollee in a managed care plan (MCP). It is often thought of
as the managed care equivalent of fee-for-service (FFS) claims. Encounter
data reflect that a provider rendered a specific service under a managed
care delivery system, regardless of if or how the MCP ultimately
reimbursed the provider. They contain substantially the same information
included on claim forms (e.g., UB-04 or CMS 1500), although not
necessarily in the same format. However, because some managed care
providers and/or services may be paid via capitation or episodes of care,
rather than based on a claim submitted for individual services rendered,
encounter data may be less complete or accurate than claim data. As
payment methodologies have begun to incorporate value-based payment
elements (such as bundled payment or episode payment), collecting
complete and accurate encounter data has become even more crucial.
Since 1999, CMS has required states to submit complete and accurate
enrollment and utilization data, including FFS claims and encounter
records, through the Medicaid Statistical Information System (MSIS). In
2011, CMS began working with state agencies and other stakeholders to
build a new data infrastructure to replace MSIS. The Transformed
Medicaid Statistical Information System (known as T-MSIS) is intended to
modernize the way states submit data about beneficiaries, providers, MCPs,
FFS claims, third-party liability, and encounters to CMS. States must

PROTOCOL FIVE

ACTIVITY 1: REVIEW STATE REQUIREMENTS

239

comply with the T-MSIS requirements and all associated guidance for all
managed care data submitted to CMS. 65
The availability of accurate and complete encounter data is important to the effective operation
and oversight of MCPs that serve enrollees covered by Medicaid and the Children’s Health
Insurance Program (CHIP) (see Box 5.1).
Box 5.1 State Uses of Encounter Data
• Develop capitation rates
• Perform risk adjustment
• Measure quality
• Implement alternative payment methods
• Conduct program integrity
• Engage in policy development

Federal regulations at 42 CFR Part 438 include several provisions related to encounter data.
•

All providers must submit claims and/or encounters to states for all services regardless of the
method by which a plan pays its providers (e.g., FFS, capitated, basis, or sub-capitation). (42
CFR 438.818(a))

•

States must review and validate encounter data on initial receipt from their MCPs, and again
when they submit it to CMS. (42 CFR 438.818(a)(2))

•

States must submit complete, accurate, and timely encounter data to CMS in a standardized
format (i.e., T-MSIS). (42 CFR 438.818(a)(3))

•

CMS may impose penalties on states for noncompliance by withholding Federal Financial
Participation (FFP) funds. (42 CFR 438.818(c))

•

This protocol provides guidance to EQROs on validating the accuracy and completeness of
encounter data submitted by MCPs.

Getting Started on Protocol 5
To complete this protocol, the EQRO undertakes five activities for each MCP (Figure 5.1).

More information about T-MSIS requirements is available at https://www.medicaid.gov/medicaid/datasystems/macbis/transformed-medicaid-statistical-information-system-t-msis/index.html.

240 | PROTOCOL FIVE

Figure 5.1. Protocol 5 Activities

Two supplemental resources are available to help EQROs validate encounter data:
•

Worksheets for Protocol 5. Encounter Data Tables, a set of worksheets that can be used to
document acceptable error rates and data element validity requirements, findings from the
review of individual encounter records, a comparison of findings to state-identified
benchmarks, results from the EQRO’s validation of medical records, and a suggested format
for reporting encounter data validation information in the EQR technical report. Format for
Reporting Encounter Data Validation Information in the EQR Technical Report

•

Appendix A. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems

In addition, it may be helpful to refer to the CMS Encounter Data Toolkit, which contains
additional information and resources about the validation process. This toolkit is available at
https://www.medicaid.gov/medicaid/downloads/medicaid-encounter-data-toolkit.pdf.
The remainder of this protocol outlines the steps associated with Activities 1 through 5.

PROTOCOL FIVE | 241

Activity 1: Review State Requirements
Activity 1 is intended to ensure the EQRO has a complete
understanding of a state’s requirements for each MCP’s
encounter data. Before initiating Activity 1, EQROs should
request all available encounter data guidance from states,
including encounter reporting requirements and standards,
data dictionary, edit checks, and other documents. At the
outset of Activity 1, the state should provide the EQRO
with at least the following information:
1. Specific requirements regarding the MCPs’
collection and submission of encounters. Some states
may formalize these requirements in contractual
language or companion guides. The state should provide
the EQRO with a detailed list of all requirements, by
plan and plan type

WORKSHEET 5.1
WORKSHEET 5.2
Resource for Activity 1
Worksheet 5.1. Specification of
Acceptable Error Rates and Identified
Areas of Concern
• Provides guidance for the EQRO’s
review of a state’s specific
requirements for reporting encounters
Worksheet 5.2. Data Element Validity
Requirements
• Provides a template for the EQRO to
document the state’s specific
requirements for validating each data
element by type of service

2. Requirements regarding the types of encounters that
must be validated (e.g., inpatient hospital, professional,
home health). The state may find it difficult to integrate
some types of encounters (e.g., non-emergency transportation or atypical providers) into its
data systems. Whenever possible, the state should direct the EQRO to alternative sources to
validate this information. Note that under this protocol, a state could direct the EQRO to
validate all of an MCP’s encounter data or a subset of an MCP’s encounters. If the state
chooses to validate a subset of the encounter data based on provider type, it must validate all
encounters for the selected provider type across all MCPs. If the state chooses to validate
encounter data for a subset of MCPs, it must validate all encounters for the subset of MCPs
3. Standards for the submitted data, including the following:

An operational definition of an “encounter,” such as adjudication status, and other
relevant details

Types of encounters MCPs must report (e.g., inpatient hospital, outpatient, professional,
home health)

Format in which encounters must be submitted (837 standard transaction, proprietary)

242 | PROTOCOL FIVE

Objective standards to which encounter data will be compared (e.g., number of
beneficiaries with at least one encounter)

4. State standards for encounter data completeness and accuracy. The state should clearly
specify acceptable rates of accuracy and completeness for each data element for each field
for each encounter type, which may depend on the intended use of the encounter data.
Although initial error rates may be higher, each MCP’s targeted error rate should be below 5
percent for each time period examined. The state should align its own standards with those
required to satisfy T-MSIS requirements
5. Data dictionary and companion guides. States often cite data dictionaries or companion
guides in managed care contract language for reference to accountability and standards for
encounter data. Those may be updated on a more regular basis than the contracts themselves.
For states that employ a fiscal intermediary, the intermediary may be the best source of this
information
6. Description of the information flow from the MCP to the state, including the role of any
contractors or data intermediaries. States that use separate organizations for medical and
behavioral health should include details about how the data are collected and integrated into a
single system, as well as challenges the EQRO may face in handling these data
7. A list and description of automated edits or checks performed on the data when received
into the state system (Medicaid Management Information System or data warehouse). This
should include information about how the system handles encounters that fail an edit check.
For example, does the system reject an entire file if one encounter is rejected?
8. The timeliness requirements for data submissions (e.g., how far from the original date of
service the record must be submitted), and standards for timeliness, as applicable and as laid
out by the state in contract documents. States are increasingly able to process high volumes
of records on a daily basis, while some prefer a monthly submission from plans. States also
may have various tolerance levels for what percentage of records must meet particular
timeliness standards
9. Any EQR validation reports from previous years. Previous reports can provide useful data
points for determining how much progress MCPs have made in improving data quality and
completeness, as well as giving a state picture of improvement or challenges over time
10. Any other information relevant to encounter data validation. States may find they use
other documentation or context in their own analyses of their MCP’s encounter data. If
supplementary information will provide relevant context to encounter validation, such as a
list of excluded providers, it should be provided to the EQRO

Activity 2: Review the MCP’s Capability
Activity 2 is intended to evaluate an MCP’s ability to collect complete and accurate encounter
data. Before assessing the output produced by the MCP’s information system, the EQRO should

PROTOCOL FIVE | 243

determine whether the system is able to collect and report high quality encounter data. To do so,
the EQRO should assess the information system in two steps (described in more detail below):
1. Review the MCP’s most recently completed Information System Capacity Assessment
(ISCA)
2. Interview MCP personnel to clarify ISCA findings as necessary

Step 1: Review the MCP’s ISCA
The EQRO should determine whether the MCP has
completed an ISCA review within the past two years. 66 If a
recent ISCA has been completed, the EQRO should review
the findings. If the MCP has not conducted an ISCA within
the previous two years, the EQRO must conduct one
consistent with the processes discussed in Appendix A.
The EQRO should review the MCP’s ISCA to identify
weaknesses in the MCP’s information systems. This
assessment determines where and how information systems
may be vulnerable to incomplete or inaccurate data capture
or processing, integration, storage, or reporting. Based on the
findings from the ISCA, the EQRO should understand the
following:
1. IT system architecture, file structure, information flow,
and data processing procedures
2. Specific programming language used by the system (e.g.,
SQL) 67
3. Process by which the MCP modifies its source code to
address changes in state reporting requirements (Note:
The EQRO should obtain all source code from the MCP)

WORKSHEET A.1
WORKSHEET A.2
WORKSHEET 5.1
Resource for Activity 2, Step 1
Worksheet 5.1. Specification of
Acceptable Error Rates and Identified
Areas of Concern
• Provides guidance for the EQRO’s
review of a state’s specific
requirements for reporting encounters
Appendix A guides the information
systems review. The appendix includes
two worksheets:
• ISCA Worksheet A.1 is the tool used
for the assessment
• ISCA Worksheet A.2 is used by the
EQRO to assess the adequacy of
MCP policies and procedures based
on the information collected in
Worksheet A.1

4. Other claims/encounter processing issues

How the system handles voids, adjustments, crossovers, and records not requiring
payments, such as for sub-capitated arrangements

SQL is programming language for storing, manipulating, and retrieving data in databases.

244 | PROTOCOL FIVE

Whether the system verifies encounters at both header and detail levels 68

Whether there are processes in place to identify “orphan” header or detail records 69

Whether there any service types (e.g., non-emergency transportation or behavioral
health) not in the system

5. Completeness of data

6. Written policies and procedures for edits and audits
7. Claims/encounter system demonstration

Whether the system permits working with the data in a “test” environment

8. Processes for merging and/or transferring data
9. Processes for encounter data intake, logging, adjudication, and denial
10. Audits performed to assure data quality and accuracy and processing timeliness
11. Maintenance and updating of provider data, including how the MCP identifies providers or
organizations excluded from the Medicaid program each month (e.g., List of Excluded
Individuals and Entities), and whether the MCP requires its provider network to update
provider data each month
12. Processing of enrollment data, including a description of how the system identifies
beneficiaries as information changes over time (e.g., how the system handles name and
address changes)
13. Specific claims and encounter verification procedures
14. Frequency of information updates (e.g., how often does the MCP update its provider table?)
15. Management of enrollment and disenrollment information
During the review of ISCA findings, the EQRO may identify encounter data errors by MCPs (see
Box 5.2). If the EQRO identifies issues that may contribute to inaccurate or incomplete
encounter data, the EQRO should list any concerns about the encounter data in Column 4 of
Worksheet 5.1. Specification of Acceptable Error Rates and Identified Areas of Concern for each
encounter type listed.

The detail-level on an encounter refers to information included on the individual lines contained within the encounter (such as
charges or procedure codes for multiple services provided within a single visit); the header-level refers to information
provided at the claim level (such as beneficiary ID, provider ID, date of service, and diagnoses).

An orphan encounter is one in which there are one or more detail records without an associated header record, or vice versa.

PROTOCOL FIVE | 245

Box 5.2 Potential Causes of Encounter Data Errors by MCPs
• Non-standard codes or forms
• Inadequate front-end data edits
• Lack of provider contractual requirements that tie payment to data submission
• Use of default dates of service or provider identifiers
• Failure to collect key demographic data elements
• Out of date or incomplete reference tables
• Failure to collect Medicare crossover claims
• Inconsistent use of adjusted and void claims

Step 2: Interview MCP Personnel
After reviewing the findings from the ISCA, the EQRO should conduct follow-up interviews
with MCP personnel as needed to supplement the information in the ISCA and ensure its
understanding of the MCP’s information systems and processes. The EQRO should refer to
ISCA components that the MCP uses to produce performance measures, including enrollment,
medical, pharmacy, provider, lab, and other ancillary or supplemental data sources.

Activity 3: Analyze Electronic Encounter Data
Activity 3 is the core function used to determine the validity of the encounter data. When the
EQRO has completed the steps within this activity, it should know whether the data are
complete, of high quality, and can be used for analysis of quality, access, program integrity
monitoring, among other critical state activities. If the EQRO cannot confirm the quality of the
data after completing this activity, it should not proceed to Activity 4, the Medical Record
Review. Instead, the EQRO should work closely with the state or plans to determine underlying
problems or acquire additional information to determine the quality and usefulness of the data
submitted. Difficulties completing this analysis may need to be summarized for the state as
indicating serious data quality issues.
The EQRO should use the information obtained from these analyses, the ISCA tool, follow-up
interviews, and the results of state edits to assess the completeness and accuracy of the MCP’s
encounter data. The results of Activity 3 will inform the development of a long-term monitoring
strategy for assessing the quality of the encounter data. As the data evolve over time, the EQRO
will be able to design targeted validation strategies to identify problem areas requiring resource
intensive medical record review.
Under this activity, the EQRO should carry out the following Steps 1 through 4.

246 | PROTOCOL FIVE

Step 1: Develop a Data Quality Test Plan Based On Data Element Validity
Requirements
The EQRO should use the information obtained from Activities 1 and 2 (including the ISCA
review and follow-up interviews with MCP staff) to develop a data quality test plan. The plan
should:
•

Account for front-end edits already built into the MCP’s data system so that it focuses on
issues that the MCP may have inadvertently missed or allowed for other reasons

•

Specify the areas to be tested and the expected results

To be of greatest use to states and other stakeholders, the EQRO should develop a plan that
addresses the following questions:
The general magnitude of missing encounter data. The EQRO should use information from
the MCP about encounters that fail front-end edits and the reasons for these failures to determine
whether, and how much, encounter data is missing. The EQRO should compare these results
with normative data on encounters for similar populations for this purpose. Examples of the use
of benchmarks for assessing encounter data completeness are available in the Encounter Data
Toolkit, available at https://www.medicaid.gov/medicaid/downloads/medicaid-encounter-datatoolkit.pdf
1. Types of encounters that may be missing. MCPs that pay for “bundled” services (e.g.,
prenatal care) or that capitate providers (e.g., for primary care) may not receive complete
encounter information from these providers. The EQRO should apply specific knowledge
about the MCP’s contractual relationships with providers to identify specific areas to look for
missing services. The EQRO should obtain information from the MCP on the use of bundled
payment and capitation to inform its plan
2. Overall data quality issues. The EQRO should identify specific data quality problems such
as inability to process or retain certain fields, or limited coding specificity on the encounter
data record
3. MCP data submission issues. The EQRO should identify problems the MCP has compiling
its encounter data and submitting the data files to the state

PROTOCOL FIVE | 247

Step 2: Encounter Data Macro-Analysis—Verification of Data Integrity
Steps 2 and 3 of Activity 3 are closely related. When the
EQRO reviews the data for accuracy and completeness, it
conducts both macro- and micro-analyses. Step 2 describes
the macro-analysis, while Step 3 describes the microanalysis.
In Step 2, the EQRO should:
•

Analyze and interpret data in specific fields

•

Check the data for volume and consistency

WORKSHEET 5.3
Resource for Activity 3, Step 2
Worksheet 5.3. Evaluation of Submitted
Fields
• Provide a template for the EQRO to
document its findings for each data
element

Without duplicating the state’s edit checks, the EQRO should analyze and interpret data in
submitted fields. In addition:
1. Is there information in the field, and is that information of the type requested?

The EQRO should check each field to determine whether its values are of the type and
size found in the state’s data dictionary, or in nationally recognized standards. For
example, if CPT®-4 codes are requested, the field should have five digits. If the state’s
Medicaid and CHIP beneficiary ID is requested, the field should contain the correct
number of letters and characters

2. Are the values valid and in the correct format?

To what extent are the values in the field valid? For instance, if ICD-10 diagnosis codes
have been requested, are the values in the diagnosis field valid for that standard?

Do critical fields contain non-missing values in the correct format and specificity (e.g.,
maximum number of characters in a diagnosis) and that values are consistent across
fields?

3. Are the data available?

Are all required data elements reported?

When viewed by date of service, are there gaps in the data?

Is beneficiary enrollment consistent over time?

When broken out by population subgroups or service types, does consistency persist?

Do the data exist for all service types?

4. Do the data meet basic consistency expectations?

Are the number of encounters consistent over time?

248 | PROTOCOL FIVE

5. Are the state’s identifiers (IDs) accurately incorporated into the MCP’s information system?

The EQRO should compare the encounter data file to the state’s eligibility file and
check for accuracy of the IDs and other eligibility information (e.g., age, sex, and
eligibility category). In addition, the EQRO should determine whether there are
encounter data for the expected proportion of beneficiaries in comparison to utilization
norms for similar populations

6. Is the information for each critical field within required ranges and is the volume of data
consistent with the MCP’s enrollment? For example, can the following types of questions be
answered with the data:

o
o

What is the rate of emergency department utilization per 1,000 member months?
What percentage of beneficiaries have at least one encounter during the year?

Note: The EQRO should automate these analyses and perform them as a standard data review
process. The EQRO should perform these analyses for each service type (e.g., inpatient hospital,
outpatient, professional, home health, durable medical equipment) and for each data field within
a service type.

Step 3: Encounter Data Micro-Analysis—Generate and Review Analytic
Reports
1. In Step 3, the EQRO should move beyond analyses
focused on data integrity and that are field-specific to
analyses that cross fields and provide a broader view
of whether the data can be used for meaningful
analyses. Often data elements may meet basic
expectations, but until multiple fields are used together
for analysis, some data quality issues may not be
detected. Examples of analytic reports that can detect
broader data quality issues are:
•

WORKSHEET 5.3
Resource for Activity 3, Step 3
Worksheet 5.3. Evaluation of Submitted
Fields
• Provide a template for the EQRO to
document its findings for each data
element

Reasonability tests

The EQRO should develop frequency distributions of the values and compare them to
normative data from similar populations to determine whether the values make sense.
For example,

If “place-of-service” is a required field, the values should be distributed across a range
of values (e.g., IP hospital, OP hospital, ED, or office)

The number of enrollees, the number of encounters, and counts and totals for various
eligibility categories or demographic subgroups, diagnoses, or types of service

Frequency distributions on specific fields, as well as on the variables created explicitly
for data validation purposes (e.g., beneficiary age from date of birth)
PROTOCOL FIVE | 249

•

Distributions on subsets of data, especially where there are specific concerns about data
validity. For example, if the EQRO finds a low rate of utilization for outpatient services,
it could analyze the data by provider zip code to determine whether the files are missing
specific zip codes, causing the system to reject records. By taking a deeper dive into the
data, the EQRO could detect a different problem than originally expected

Univariate statistics (e.g., means, medians, quartiles, and modes) as appropriate.
The EQRO should check the output of these reports for reasonableness and to detect
specific problems such as entire categories of data missing from the regular data
submissions

Analyses by dates of service versus adjudication dates

•

Checks by provider types

•

Analyzing the data by dates of service and by adjudication dates can detect issues in
consistency over time. Inconsistent processing can indicate other problems within the
MCP’s IS system, which may impact data validity. After establishing the length of time
between service and adjudication dates, the EQRO should compare them with standards
or benchmarks for data submission and processing
The EQRO should review the data by provider type to identify missing provider types
and examine fluctuations in patient visits by provider type for specified time periods.
The EQRO should compare the distribution of encounters by provider type to normative
information. The EQRO should also examine diagnosis or procedure codes by provider
type to ensure that the relationship between provider specialty and the services rendered
is consistent

Relational analyses by service type or episodes of care

The relationship between ancillary services (e.g., labs, x-rays, etc.) and visits

The relationship between primary and specialty care visits

Grouped services expected in particular types of visit or episodes of care

The relationship between outpatient visits and the number of prescriptions dispensed

Outpatient services associated with inpatient admissions

Other relationships between service types previously identified as problematic through
the ISCA, front-end edits, or other EQRO validation activities.

Analyses broken out by demographic group or subpopulation

If not addressed already in the MCP’s front-end edits, the EQRO should conduct
analyses that take a beneficiary’s age and gender into account. For example, the EQRO
could verify that a gender-specific diagnosis (e.g., endometriosis) or procedure (e.g.,
caesarean delivery) is consistent with the beneficiary’s age and gender derived from the
encounter header record or the beneficiary’s enrollment record

250 | PROTOCOL FIVE

•

Analytic questions

The EQRO should use information gathered in previous steps to select a question or
series of questions it might answer using the encounter data as another step in
determining quality and usability. For instance, the EQRO could take a particular
measure of interest to the state and replicate it across all MCPs or within MCPs, such as
the number of beneficiaries per primary care provider.

The EQRO should conduct these analyses on the encounter data and compare the results to
external benchmark information (Step 4). As part of the review, the EQRO should display the
data quality findings graphically to identify issues for further investigation and to communicate
the results of the data quality review. The EQRO should generate these reports for each MCP
and on the entire encounter data set for all MCPs to account for problems associated with small
numbers of encounters for individual MCPs. For examples of these types of displays, see Section
5 of the CMS Encounter Data Toolkit. 70

Step 4: Compare Findings to State-Identified Benchmarks
In this step, the EQRO compares the encounter data
submitted by each MCP to benchmarks identified by the
state. The EQRO will need to identify and document
these benchmarks. The benchmarks can be obtained
from various sources, including:

WORKSHEET 5.4
Resource for Activity 3, Step 4
Worksheet 5.4. Benchmark Utilization
Rates

•

Aggregate encounter data from all Medicaid or
CHIP MCPs in the state

•

Historical FFS or PCCM data

•

Other comparable states

•

The CMS Encounter Data Toolkit

•

Other benchmarks, such as MCP financial reports, commercial MCPs, national standards,
HEDIS®, or the Child and Adult Core Set measures 71

• Provides a template for the EQRO to
compare findings to state-identified
benchmarks

The EQRO should understand which differences from comparison data require further
investigation. For example, emergency room utilization might be lower in managed care than in
FFS. However, large swings in utilization from one time period to the next, or differences from
the benchmark that are not explained by delivery system differences may indicate incomplete or

The CMS Encounter Data Toolkit is available at https://www.medicaid.gov/medicaid/downloads/medicaid-encounter-datatoolkit.pdf.

HEDIS® benchmarks are published annually by NCQA in The State of Healthcare Quality Report, available at
http://www.ncqa.org/report-cards/health-plans/state-of-health-care-quality. Child and Adult Core Set benchmarks are available
at https://www.medicaid.gov/medicaid/quality-of-care/index.html.

PROTOCOL FIVE | 251

erroneous encounter data, rather than a difference in provider practice patterns. The EQRO
should vet its assumptions about changes in utilization with the MCPs and the state to determine
what follow-up analyses might be required. For example, unusual changes in utilization and
outcomes may occur after a natural disaster (such as a hurricane). The EQRO should discuss
anomalous findings with the state to assess underlying factors that may utilization or outcomes.

Activity 4: Review Medical Records
The purpose of Activity 4 is to confirm the findings from the
analysis of encounter data performed in Activity 3, using
retrospective reviews of patient medical records. This
activity makes the following assumptions about the record
review:

WORKSHEET 5.5
WORKSHEET 5.6
Resources for Activity 4

1. Reviews performed under the guidance of this protocol
and activity should be independent of record reviews
performed for all other purposes, including those
performed to validate performance measures, for
program integrity, etc.

Worksheet 5.5. Medical Record Review
for Encounter Data Validation

2. The state will determine the timing and frequency of all
medical record reviews

• The Data Field Validation Table
indicates whether codes or other data
fields in the encounter record match
the medical record, and vice versa

3. Once the state has determined a review of medical
records is appropriate, the EQRO will draw a sample of
records for validation on a regular and periodic basis, as
directed by the state agency

• The Event Validation Table indicates
whether an encounter record
matches the medical record, and vice
versa

Worksheet 5.6. Medical Record Review
Results Summary Sheet
• Summarizes the results of the
medical record review, including the
error rate and reasons for errors

EQROs should approach the validation of encounters from
medical records as if they are research questions with clear
hypotheses, well-defined sampling methodology, and predetermined error tolerances. Questions
under consideration for medical record review generally fall into the following categories:
1. Questions of Description

Are all of the diagnosis codes in the patient’s medical record on the associated
encounter?

Are all of the procedure codes in the patient’s medical record on the associated
encounter?

Does the Date of Birth (DOB) listed in the beneficiary’s medical record match the DOB
found on the encounter header?

2. Questions of Relationship

Are there differences in the number of diagnoses reported for women compared to men?

252 | PROTOCOL FIVE

Are there differences in the distribution of Evaluation & Management (E&M) procedure
codes by age group?

Are there differences in the utilization of specific procedure codes by geographic area
(e.g., county)?

3. Questions of Comparison

Are there differences in the average number of diagnoses coded on the encounter records
compared to those found on FFS or PCCM claims?

Are there differences in the distribution of E&M procedure codes (i.e., 99201 – 99205)
on the encounter records compared to those found on FFS or PCCM claims?

Are there differences in the distribution of Place of Service codes on the encounter
records compared to those found on FFS or PCCM claims?

EQROs should limit each medical record review to a specific encounter type (e.g., inpatient
hospital admissions, physician office visits). The EQRO should ensure that in narrowing the
scope of the review, it does not overlook service types that are vulnerable to undercounting (such
as prenatal and postpartum visits).
EQROs should determine the sample size for the medical record review using standard sampling
methodology. The sample size will depend in part on the minimum error rate the EQRO must
detect and the number of subpopulations for which validation is conducted. Note that it is not
appropriate to substitute a record that is missing. Substitution may be allowed if a medical record
is out of the office for legal review. Box 5.3 provides sampling guidance for medical record
reviews.
Box 5.3 Sampling Guidance for Medical Record Review
• See Appendix B for an overview of sampling approaches and guidance for calculating sample sizes
• Set sample sizes for medical record review sufficient to estimate the error rate for each type of encounter within
each population, with equal precision for each time period under review
• It may be appropriate to allow the substitution of a medical record if it is out of the office for legal review. However, it
is not appropriate to substitute a record that is missing
• A statistician or other staff with expertise in sample design and implementation should advise the state and/or
EQRO on the appropriate sampling strategy for the medical record review

Once the sample of medical records is selected, the EQRO needs to request the medical records
from providers, compare the content of the encounter records and medical records, and document
findings. The state should provide written guidance to the EQRO about the procedures for
conducting the medical record review, including the reporting requirements, the data elements
chosen for validation, and the error categories used. The EQRO should employ experienced
clinical coders to review codes based on the diagnoses stated by the provider in the patient’s
medical record.

PROTOCOL FIVE | 253

To obtain medical records for review, the EQRO should give each provider a list with each
patient’s name, age, and sex, the provider’s name, and the target dates of service. This
information should be sufficient for the provider to identify the beneficiary and locate the correct
record. Guidelines should describe exactly how to document the findings of medical record
review and should include:
•

Directions for reviewing medical records

•

Instructions for evaluating conflicting documents

•

Instructions on what to do when no code can be readily assigned

•

Use of optional codes

•

Definitions of what constitutes an “error”

•

Lists and locations of approved reference materials

•

Whom to consult for additional assistance

In defining what constitutes an error, the state should consider the following:
•

Designate certain errors as “critical” depending on the intended use of the data. These
designations may evolve over time as encounter data issues change. For example, the initial
stages of analysis may focus on diagnosis (ICD-10) and procedure (CPT®) codes rather than
provider specialty or place of service codes. The latter two fields may be of little value if the
former fields are inaccurate

•

Distinguish error “tiers” (e.g., critical, serious, moderate), which may permit use of encounter
data that may be incomplete or have some inaccuracies

Activity 5: Submit Findings
After the completion of Activities 1 through 4, the EQRO
should create data tables that display summary statistics for
the information obtained from each MCP. Summarizing the
information in tables makes it easier to evaluate the
findings and highlight patterns in the accuracy and
completeness of the data. The EQRO should draft a
narrative to accompany the tables, highlighting individual
MCP issues and providing recommendations to each MCP
and the state about improving the quality of the encounter
data.

WORKSHEET 5.7
Resource for Activity 5
Worksheet 5.7. Suggested Format for
Reporting Encounter Data Validation
Information in the EQR Technical Report
• Provides a template for reporting
results from the encounter data
validation activity in the EQR
technical report

In its findings and recommendations, the EQRO should assess the MCP’s ability to provide the
state with encounter data that meets the quality standards for submission to the state for use in TMSIS. The EQRO should also assess the MCP’s ability to produce reliable and valid
performance measures as specified in the managed care quality strategy.
END OF PROTOCOL 5
254 | PROTOCOL FIVE

Worksheets for Protocol 5: Encounter Data Tables
Instructions. Use these or similar worksheets as a guide when validating encounter data. The encounter data
worksheets 5.1 through 5.4 are intended to help document acceptable error rates and data element validity
requirements, findings from the review of individual encounter records, and comparison of findings to state-identified
benchmarks. Worksheet 5.5. Medical Record Review for Encounter Data Validation is intended to help record results
from the EQRO’s validation of medical records. Worksheet 5.6. Medical Record Review Results Summary Sheet
summarizes the results of the medical record review, including the error rate and reasons for errors.
Worksheet Name
Worksheet 5.1. Specification of Acceptable Error Rates
and Identified Areas of Concern

Protocol Activity and Step
Activity 1. Step 1. Review the MCPs’ ISCA

Worksheet 5.2. Data Element Validity Requirements
Worksheet 5.1. Specification of Acceptable Error Rates
and Identified Areas of Concern

Activity 2. Step 1. Review the MCP’s Capability

ISCA Worksheet A.1. ISCA Tool
ISCA Worksheet A.2. ISCA Worksheet & Interview Guide
Worksheet 5.3 Evaluation of Submitted Fields

Activity 3. Analyze Electronic Encounter Data

Worksheet 5.4. Benchmark Utilization Rates
Worksheet 5.5. Medical Record Review for Encounter
Data Validation

Activity 4. Step 1. Obtain and Review Medical Records
and Document Findings

Worksheet 5.6. Medical Record Review Summary Sheet
Worksheet 5.7. Suggested Format for Reporting
Encounter Data Validation Information in the EQR
Technical Report

255 | PROTOCOL FIVE

Activity 5. Submit Findings

Worksheet 5.1. Specification of Acceptable Error Rates and Identified Areas of Concern
Instructions. Worksheet 5.1 provides guidance for the EQRO’s review of a state’s specific requirements for reporting
encounters. The EQRO should add rows as necessary to include all service types used by the state. Definitions for
this activity are as follows:
•

Encounter Data Error Types
ο

Missing. A service rendered for which there is no encounter record

Surplus. An encounter submitted for a service that was never rendered, or which duplicates another record

Erroneous. Services rendered where there is an error in the encounter record

•

Acceptable Error Rate. For each type of service (e.g., inpatient) and error (e.g., missing), the EQRO should
document the state’s acceptable error rate. In Worksheet 5.1, the acceptable error rate column expresses this
rate as the percentage of missing, surplus, or erroneous records the state will accept from the MCP. Files with an
error rate exceeding the thresholds are unacceptable. For example, a state might set error thresholds for office
visits at less than 10 percent for missing encounters, less than 2 percent for surplus encounters, and less than 5
percent for encounters with erroneous information. If the state expresses its error tolerance in a different way, the
EQRO should adjust the acceptable error rate accordingly.

•

Areas of Concern. Based on the ISCA and other information, the EQRO should identify errors that it reasonably
expects might occur. The EQRO could derive this information from work it performs in Protocol 2, Validation of
Performance Measures Reported by the MCP. It should use the information to guide subsequent reviews.
Service type

Acceptable
error rate

Error type

Office visit – includes all services,
except dental and mental health /
substance abuse

Missing

Surplus

Erroneous

Office visit – includes mental health /
substance abuse services only

Missing

Surplus

Erroneous

Missing

Surplus

Erroneous

Inpatient admission – includes all IP
services, except mental health /
substance abuse services

Missing

Surplus

Erroneous

Inpatient admission – includes
mental health / substance abuse
services only

Missing

Surplus

Erroneous

Other types of encounters (e.g.,
emergency department, lab / x-ray,
pharmacy, physical therapy)

Missing

Surplus

Erroneous

Office visit – includes dental services
only

Area of concern (Yes/No/Describe)

256 | PROTOCOL FIVE

Worksheet 5.2. Data Element Validity Requirements
Instructions. The EQRO should document clearly the state’s specific requirements for validating each data element
by type of service. The EQRO should then evaluate each file received to validate the specific data elements. The
EQRO should add rows as necessary to incorporate all data elements for which the state identifies specific validation
requirements. Definitions for this activity include:
•

Expectation. The EQRO should use this column to describe the general requirement(s) for validating each data
element.

•

Validation Criteria. The EQRO should use this column to document the validation threshold for each data
element. Typically, the column will include a quantitative expression of the description in the Expectation column.
Examples are shown in Worksheet 5.2.

Service type:
Data element

Expectation

Examples of validation criteria

Enrollee ID

A valid member ID (e.g., Medicaid ID
Number; SSN) as documented in the
state’s eligibility file.

98% valid.

Date of service (FDOS;
LDOS)

Dates of service should be distributed
across the entire period analyzed. Look for
large month-to-month increases or
decreases. Also, look for months with
encounters that may be missing entirely.

Calculate the average number of
encounters per month over the period
specified in the study. In general,
month-to-month differences should be
relatively small. Document any
outliers and request an explanation
from the MCP.

Unit of service (Quantity)

This field should generally include the units X% non-zero.
billed for each type of medical service (e.g.,
< Y% should be 1 if CPT® code in
2 units ≥ 23 minutes through 37 minutes).
range 99200-99215, 99241-99291.

Procedure code

Should include valid CPT® and HCPCS
values, or another state-approved code.

At least 98% of the values in this field
should be valid (i.e., non-zero, not
blank, and not 8-or-filled) and in the
expected format.

Revenue code (Hospital)

If the facility uses a UB04 claim form, this
field should always be populated on
inpatient encounters.

At least 98% of the values for this
field on inpatient claims should be
valid and in the expected format.

PROTOCOL FIVE | 257

Worksheet 5.3. Evaluation of Submitted Fields
Instructions. As the EQRO reviews encounter records, it should document its findings for each data element on
Worksheet 5.3 or a similar form. To complete Worksheet 5.3 or similar tool, ask the following questions:
1. Is there information in the field, and is the information of the type (e.g., numeric) and format (e.g., MM/DD/YYYY)
required?
•

The EQRO should check each data element to determine whether its values are of the type and size specified
in the state’s data dictionary. For example, if CPT®-4 codes are required, the field should have 5 characters. If
the state’s Medicaid and CHIP beneficiary ID is required, the field should include the specified number of
alpha, numeric, or alphanumeric characters.

2. Compared to a generally accepted external standard, are the values in the specified field valid? For example, do
the values in the field PROC-CODE match those found in the ICD-10-CM tables?
Field is populated
Required field

Correct type

Correct size

Value is valid

Member ID
Plan ID
Billing provider ID
Rendering
provider ID
Primary diagnosis
code
Primary procedure
code
First date of
service
Last date of
service
Quantity (units)
Add rows as
needed
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

258 | PROTOCOL FIVE

Worksheet 5.4. Benchmark Utilization Rates
Instructions. The EQRO should use this worksheet to compare its findings to state-identified benchmarks. Revise
the column headings to reflect the specific benchmarks identified by the state. EQROs should add measures as
specified by state validation requirements. CMS suggests that eligibility measures, if included, should align with the
eligibility group code in T-MSIS.

Measure

MCP rate

FFS/PCCM
rate

Comparable
state(s) rate(s)

Other
comparable
rate (specify)

Inpatient discharges
Inpatient LOS
Overall
By high-volume MS-DRGs
By eligibility category/cohort
Ambulatory surgeries
Total number of surgeries
By high-volume CPTs® or ambulatory
surgery categories
Total surgeries (per 1,000 members)
By high-volume CPTs® or ambulatory
surgery categories
Providers
Primary care physicians
Specialists
Other (e.g., mental health providers)
Enrollees
Total number of enrollees
By eligibility category
By age, gender categories
Service utilization
Total number of service users
By eligibility category
By age, gender categories
Visits
Total number of visits
Average visits per enrollee
Average visits per user
By visit type (e.g., well-child)

PROTOCOL FIVE | 259

Measure

MCP rate

FFS/PCCM
rate

Comparable
state(s) rate(s)

Other
comparable
rate (specify)

Other service types (e.g., Rx)
Total number by service type
Encounters by enrollee/service type
Encounters by enrollee/service type
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

260 | PROTOCOL FIVE

Worksheet 5.5. Medical Record Review for Encounter Data Validation
Instructions. Complete Worksheet 5.5 for each record in the sample. Transfer results from the Event Validation and
the Data Field Validation tables to Worksheet 5.6. Medical Record Review Results Summary Sheet.
Reviewer name:

Review completion date:

Data element

Field value

Data element

Medical record ID number

Practice name

Patient name

Practice TIN

Patient ID number

Rendering provider name

Patient gender

Rendering provider PIN

Patient date of birth

Primary diagnosis

First date of service

Principal procedure

Field value

Last date of service

Event Validation Table
Line number

Procedure

Event noted on
encounter record

Event noted in
medical record

Match

No match

1
…
…
…
N
Note: Include one line for each procedure in the record for selected date.
If no match is found (i.e., the event is missing from either the medical record or the encounter record), record the
results on the Medical Record Review Results Summary Sheet below, and stop.
If the event is present on both the medical and encounter records, proceed to validation of the specified data fields.
Required Review: (Check One)
[

] Office visit: Includes all services, except dental and mental health/substance abuse

[

] Office visit: Includes mental health/substance abuse services only

[

] Office visit: Includes dental services only

[

] IP admission: Includes all IP services, except mental health/substance abuse services

[

] IP admission: Includes mental health/substance abuse services only

[

] Other types of encounters utilized by the state (e.g., lab/x-ray; physical therapy)

[

] Specify other service type: _______________________

PROTOCOL FIVE | 261

Data Field Validation Table
Diagnosis codes and descriptors
Encounter line #

Encounter Dx
code

Dx description

Medical record
Dx code

Match

No match

Procedure
description

Medical record
procedure code

Match

No match

Revenue
description

Medical record
revenue code

Match

No match

1
…
N
Procedure codes and descriptors
Encounter line #

Encounter
procedure code

1
…
N
Revenue codes and descriptors
Encounter line #

Encounter
revenue code

1
…
N
Note: The EQRO should edit this table to include all data elements under review.
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

262 | PROTOCOL FIVE

Worksheet 5.6. Medical Record Review Results Summary Sheet
Research question:
_____________________________________________________________________________________________
_____________________________________________________________________________________________
_____________________________________________________________________________________________
Sample size: ___________
Sampling methodology: __________________________________________________________________________
Please summarize how the MCP addresses medical record review auditing or accuracy checks:
_____________________________________________________________________________________________
_____________________________________________________________________________________________
Record of substitutions (list substitutions and reasons):
Original Record

Replacement Record

Replacement Reason

1.
2.
3.
Results:
Record
numbers
reviewed

Event noted on encounter
record

Event recorded
in medical
record

Match? (Yes / No)

Notes / Comments

1
2
3
…
N
Error rate (total records with errors/total records in sample): ____________
Reviewer summary of findings (including reasons for errors):

PROTOCOL FIVE | 263

Worksheet 5.7. Suggested Format for Reporting Encounter Data Validation Information in
the EQR Technical Report
Instructions. Use Worksheet 5.7 as a framework to report findings from the encounter data validation activities in
Protocol 5 by MCP.
For each MCP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
EQRO interview date
Type of delivery system (check all that apply)

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM □ LTSS
□ Other: specify ___________________________________

Programs (please check)
Note:

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

CHIP= Children’s Health Insurance Program; IPA = Independent Practice Association; LTSS = Long-Term
Services and Supports; MCP= Managed Care Plan; MCO = Managed Care Organization; PAHP= Prepaid
Ambulatory Health Plan; PIHP = Prepaid Inpatient Health Plan; PCCM = Primary Case Management.

Encounter Type

Records Received
and Reviewed

Total Elements
Possible

Total Matched
Elements

Percentage of
Matched
Elements

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

Inpatient
Outpatient
Office visit
Total

Inpatient Encounter
Type
Match
No Match
Total Elements
Match Percent

Outpatient
Encounter Type
Match

264 | PROTOCOL FIVE

Outpatient
Encounter Type

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

No Match
Total Elements
Match Percent

Office Visit
Encounter Type
Match
No Match
Total Elements
Match Percent

No Match for Diagnosis Code Element

Encounter Type

Total Elements

Lack of Medical Record
Documentation

Incorrect Principal
Diagnosis (Inpatient) or
Incorrect Diagnosis
Codes

Inpatient
Outpatient
Office Visit
Total

No Match for Procedure Code Element

Encounter Type

Total Elements

Lack of Medical Record
Documentation

Incorrect Principal
Diagnosis (Inpatient) or
Incorrect Diagnosis
Codes

Inpatient
Outpatient
Office Visit
Total

PROTOCOL FIVE | 265

No Match for Revenue Code Element

Encounter Type

Total Elements

Lack of Medical Record
Documentation

Incorrect Principal
Diagnosis (Inpatient) or
Incorrect Diagnosis
Codes

Inpatient
Outpatient
Total
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

END OF WORKSHEETS FOR PROTOCOL 5

266 | PROTOCOL FIVE

Protocol 6. Administration or Validation of
Quality of Care Surveys
AN OPTIONAL EQR-RELATED ACTIVITY
SECTION I. ADMINISTERING A SURVEY
ACTIVITY 1: IDENTIFY THE SURVEY PURPOSE, OBJECTIVES, AND
ACTIVITY 2: DEVELOP A WORK PLAN
ACTIVITY 3: SELECT THE SURVEY INSTRUMENT
ACTIVITY 4: DEVELOP THE SAMPLING PLAN
ACTIVITY 5: DEVELOP A STRATEGY TO MAXIMIZE RESPONSE
ACTIVITY 6: DEVELOP A QUALITY ASSURANCE PLAN
ACTIVITY 7: IMPLEMENT THE SURVEY ACCORDING TO THE WORK PLAN
ACTIVITY 8: PREPARE AND ANALYZE SURVEY DATA AND PRESENT
RESULTS IN A FINAL REPORT
SECTION II. VALIDATING A SURVEY
ACTIVITY 1: REVIEW THE SURVEY PURPOSE, OBJECTIVES, AND
AUDIENCE
ACTIVITY 2: REVIEW THE WORK PLAN
ACTIVITY 3: REVIEW THE RELIABILITY AND VALIDITY OF THE
SURVEY INSTRUMENT
ACTIVITY 4: REVIEW THE SAMPLING PLAN
ACTIVITY 5: REVIEW THE ADEQUACY OF THE RESPONSE RATE
ACTIVITY 6: REVIEW THE QUALITY ASSURANCE PLAN
ACTIVITY 7: REVIEW THE SURVEY IMPLEMENTATION
ACTIVITY 8: REVIEW THE SURVEY DATA ANALYSIS AND FINAL REPORT

PROTOCOL SIX

AUDIENCE

Background
Surveys are an important resource for assessing the experience of managed care
enrollees and providers. Information derived from surveys can help states and
managed care plans (MCPs) create a person-centered health care environment
for those enrolled in Medicaid and the Children’s Health Insurance Program
(CHIP). Enrollee surveys can be used to assess experience with their health plan
and its providers, and the quality of care they receive. Provider surveys can be
used to assess the characteristics of providers and practices that serve Medicaid
267

and CHIP enrollees, their accessibility and availability, and their experience with the Medicaid
and CHIP programs.
This protocol provides guidance for administering and validating consumer or provider surveys.
These surveys may be administered by states or MCPs (or their vendors) and validated by an
external quality review organization (EQRO) or administered by the EQRO on behalf of a state
or MCP. Because this protocol may be used for a variety of purposes, it does not specify one
survey instrument, sampling method, or analytical approach.
An overarching goal of this protocol is to provide guidance about designing and conducting
surveys that produce valid and reliable results. In this context, validity refers to surveys that
measure what they were intended to measure. Reliability refers to the internal consistency of a
survey and the reproducibility of survey results when administered under different conditions
(e.g., by different people or at different times). Please refer to the Technical Appendix at the end
of this protocol for further discussion about potential sources of survey error that can affect the
overall quality of a survey.

Getting Started on Protocol 6
This protocol includes eight activities related to administering and validating a survey (Figure
6.1). When an EQRO validates a survey, the activities in 6.1 focus on ensuring the survey was
administered correctly. Although the focus of the EQR-related activity will differ depending on
whether the EQRO’s role is to administer or validate a survey, these eight activities are common
to both roles.

268 | PROTOCOL SIX

Figure 6.1. Protocol 6 Activities

Note:

These activities pertain to survey implementation. Survey validation activities involve reviewing the
adequacy of survey implementation.

PROTOCOL SIX | 269

Two supplemental resources are available to help EQROs administer and validate a survey:
•

•

Worksheets for Protocol 6. Survey Administration and Validation Tools, which can be used
to guide the EQRO’s activities as follows:

For survey administration: Use the worksheets to track and document steps performed in
designing and implementing the survey. In the “Comments” column, document
decisions or findings

For survey validation: Use the worksheets to track and document steps performed in
validating the survey. In the “Comments” column, document the outcome of validation
activities, including sources reviewed. The worksheets can also be used as an outline for
the final report to the state. Expand the tool to include other activities or findings as
needed

Appendix B. Sampling Approaches for EQR Data Collection Activities, which provides an
overview of sampling methods

Section I of this protocol describes the activities associated with administering a survey. Section
II describes the activities associated with validating a survey.

270 | PROTOCOL SIX

Section I. Administering a Survey
Activity I.1: Identify the Survey Purpose, Objectives, and
Audience
The first step in developing a survey is to identify the
survey purpose, objectives, and audience (Worksheet
6.1). The EQRO should develop a clear understanding of
how a state will use the survey results, including what the
state wants to learn from the survey and what it plans to
do with the results (see Box 6.1).
The state should also specify the audience for the
findings, since the survey content, analysis plan, and
report format will vary based on the audience. Such
audiences and uses could include the following:

WORKSHEET 6.1
Resource for Activity 1
Worksheet 6.1. Survey Purpose,
Objectives, and Audience
• Provides a set of questions to assess
the clarity of the survey purpose,
objectives, and audience

•

Enrollees and their families. Increasingly, consumers rely on survey information to inform
their choice of health care options. To support this use, the survey design must allow for
comparisons among MCPs, potentially controlling for or stratifying enrollee characteristics

•

MCPs and providers. To promote value-based purchasing in Medicaid and CHIP,
information on the quality of care provided by MCPs and providers can be used to identify
higher- and lower-performing plans and practices and support quality improvement
initiatives

•

State policymakers. With increasing recognition of the link between better care, better
health, and more affordable care, information on enrollee experiences in managed care,
barriers to care, and the role of social determinants can be used to develop initiatives to
reduce disparities and improve outcomes
Box 6.1 Examples of Survey Uses
• Monitor and evaluate access, timeliness, and quality of care provided to Medicaid and CHIP enrollees
• Inform value-based purchasing and quality improvement initiatives
• Provide information to help Medicaid and CHIP enrollees make informed choices among MCPs

Next, the state should specify survey domains that align with the intended use of the survey
results. For example, if the survey results will be used to help enrollees choose a health plan,
specific measurement domains might include experience with the primary care provider, access
to specialty care, and treatment planning, among others.
Finally, the state should specify the unit of analysis, including populations or subpopulations of
interest. Depending on the purpose of the survey, the unit of analysis could be the entire
managed care population in the state or it could be targeted to subpopulations, for example,
PROTOCOL SIX | 271

individual MCPs, provider groups, children with chronic conditions, new Medicaid enrollees, or
individuals recently disenrolled from an MCP. This information is used to develop the sampling
approach, instrument design, and analysis plan.

Activity I.2: Develop a Work Plan
After determining the intended use of the survey in
WORKSHEET 6.2
collaboration with the state, the EQRO should prepare a
work plan that will govern the implementation of the survey
Resource for Activity 2
(including the project management plan, schedule, and
Worksheet 6.2. Work Plan
reporting requirements). Key issues to address in the work
plan are summarized in Worksheet 6.2. Refer to Activity I.6
• Provides a set of questions to assess
the work plan
for examples of typical weekly data collection schedules.
The EQRO should obtain state approval of the work plan
before implementing the survey and then administer the survey in accordance with the approved
work plan.

Activity I.3: Select the Survey Instrument
The state’s choice of a survey instrument should be
consistent with the purpose of the data collection, unit of
analysis, and goal of collecting valid and reliable data
(Worksheet 6.3). This protocol describes three options for
selecting the survey instrument:
•

Option 1. Use an existing validated survey instrument

•

Option 2. Adapt an existing survey instrument with
additional state-specific questions

•

Option 3. Develop a new survey instrument

WORKSHEET 6.3
Resource for Activity 3
Worksheet 6.3. Survey Instrument
• Provides a set of questions to assess
the selection of the survey instrument

The state may choose among the three options independently or in consultation with the EQRO.
However, there often are trade-offs in selecting an instrument. Use of an existing instrument may
provide the greatest assurance of validity and reliability but omit certain key domains of interest.
In contrast, development of a new survey instrument may provide the closest alignment with the
intended use of the survey results but validity and reliability may be untested. Thus, another
option is to adapt an existing survey instrument by adding state-specific questions to address
gaps in survey content. These three options are summarized in more detail below.
•

Option 1. Use an Existing Validated Survey Instrument

Use of an existing well-validated instrument offers several benefits such as:

Cost efficient

Readily accessible

272 | PROTOCOL SIX

Minimal development and testing hours

Available benchmark data that can be used for context and comparisons

Often translated into Spanish or other languages

Potential for rapid launch of data collection to investigate time-sensitive issues

The state or EQRO can select from a variety of existing survey instruments. Table 6.1 provides
examples of instruments that have been used to gather (1) beneficiary feedback about
experiences with health care, and (2) provider feedback on organizational issues.
Table 6.1. Examples of Existing Validated Survey Instruments
Example

Description

Beneficiary surveys
CAHPS®

• Developed by AHRQ in collaboration with the CAHPS® Consortium, the CAHPS®
survey instruments and reporting formats have undergone rigorous testing for reliability
and validity
• States frequently use the CAHPS® surveys to assess enrollees’ experiences with
managed care; versions include a Health Plan Survey; Clinician & Group Survey;
Hospital Survey; and Cancer Care Survey. An overview is available at
https://www.ahrq.gov/cahps/index.html
• Includes surveys for Child and Adult Medicaid and CHIP enrollees available at
https://www.ahrq.gov/cahps/surveys-guidance/hp/index.html
• Allows for the addition of supplemental questions; see
https://www.ahrq.gov/cahps/surveys-guidance/item-sets/search.html
• National and regional benchmarks are available; information about the CAHPS
database is available at https://www.ahrq.gov/cahps/cahps-database/index.html
• Note that there are two versions of CAHPS (AHRQ and NCQA); more information about
the differences between the two versions is available at
https://www.ahrq.gov/cahps/surveys-guidance/hp/about/NCQAs-CAHPS-HPSurvey.html
• A CAHPS Health Plan survey fielding guide is available at
https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/hp/fielding-thesurvey-hp50-2013.pdf

CAHPS Behavioral
Health Care Surveys

• Two CAHPS products are available to assess patient experience with behavioral health,
mental health, or substance use services: (1) supplemental items for the CAHPS Health
Plan Survey and Clinician & Group Survey and (2) CAHPS Experience of Care and
Health Outcomes (ECHO) Survey
• More information about CAHPS mental health surveys is available at
https://www.ahrq.gov/cahps/surveys-guidance/echo/index.html

PROTOCOL SIX | 273

Example
Long-term Services
and Supports (LTSS)
Surveys

Description
• Standardized instruments for measuring experiences with LTSS and home and
community-based services (HCBS) include the HCBS CAHPS, National Core Indicators,
and National Core Indicators-Aging and Disabilities
• More information about HCBS CAHPS is available at
https://www.medicaid.gov/medicaid/quality-of-care/quality-of-care-performancemeasurement/cahps-home-and-community-based-services-survey/index.html
• More information about National Core Indicators is available at
https://www.nationalcoreindicators.org/
• More information about National Core Indicators-Aging and Disabilities is available at
http://www.advancingstates.org/initiatives/national-core-indicators-aging-and-disabilities

Provider and practice surveys
Patient Centered
Medical Home
Assessment
(PCMH-A)

• Standardized practice-level survey instrument
• Designed to help practices monitor their progress as they transition to a medical home
care model and identify areas for improvement
• More information is available at
http://www.safetynetmedicalhome.org/sites/default/files/PCMH-A.pdf
• A modified PCMH-A survey was developed for the Comprehensive Primary Care
Initiative (PCPI) evaluation; for more information see
https://innovation.cms.gov/Files/reports/cpci-evalrpt2.pdf

Staff Experience
Survey

• Developed by the University of Chicago to assess staff experience across multiple
domains: access to care and communication with patients, tracking data, electronic
medical record, care management, quality improvement, work satisfaction, work
environment, work activities, and demographics
• Survey instrument is available at
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3752653/

Even when using an existing instrument, the EQRO should review the instrument’s reliability
and validity based on published or unpublished documentation (see Box 6.2). For example,
existing validated survey instruments may not have been validated in a Medicaid or CHIP
population or not tested in languages other than English. Selecting instruments not validated in
the target population may not yield valid or reliable results for that population. When using an
existing survey instrument, the EQRO should document findings related to reliability and
validity testing of the survey instrument, preferably in a comparable population.

274 | PROTOCOL SIX

Box 6.2 Validity and Reliability
Validity refers to the degree to which the survey is measuring what was intended to be measured and is made up of
two components:
• Face validity refers to the degree to which the survey is measuring what was intended to be measured
• Content validity refers to whether the survey questions accurately represent the concept or subject matter being
measured
Reliability refers to:
• Internal consistency of a survey
• Reproducibility of survey results when administered under different conditions (such as by different people or at
different times)

•

Option 2. Adapt an Existing Survey

Another option is to adapt an existing survey by adding or deleting items, modifying
questions, or using only certain groups of questions relevant to the state’s survey
objectives. Modifying an existing questionnaire provides the state with the flexibility to
add or change the survey content while providing many of the advantages of using a preexisting questionnaire. However, adding, deleting, or modifying questions may
undermine the validity and reliability of the questions, as well as the survey overall.
Validated questionnaires are tested “as a whole,” and modifications can change the focus
and purpose of the questionnaire.

Some surveys, such as the CAHPS® Medicaid Health Plan Survey, provide optional
supplemental questions the state can consider using to customize the questionnaire. This
has the advantage of providing a validated instrument that allows comparisons, while
accommodating special questions of particular interest to the state or MCPs.

When the EQRO adapts an existing questionnaire, it should consult with an expert in
survey design about incorporating the modification and conducting appropriate tests for
reliability and validity. Any new translations should also be tested.

Option 3. Develop a New Survey Instrument

The state or EQRO may also decide to develop a new survey instrument when the
survey purpose requires answers to questions not measured by existing instruments. A
well-designed instrument can capture information of interest and relevance to the
questions under study.

Box 6.3 includes best practices in questionnaire and design that states or EQROs should follow.
If possible, the state or EQRO should involve a survey design expert to address issues associated
with respondent burden, comprehension, and readability.

PROTOCOL SIX | 275

Box 6.3. Best Practices in Questionnaire Design
• Questions are worded clearly and briefly, and in an unbiased manner so respondents can readily understand key
terms and concepts
• Questions request information that respondents can reasonably be expected to report
• Question response categories are appropriate, mutually exclusive, and reasonably exhaustive given the intent of the
questions
• Questions are accompanied by clear, concise instructions and probes so that respondents will know exactly what is
expected of them
• All questions can be easily understood by someone with a sixth-grade reading level

In addition, the state or EQRO should work with a survey design expert to assess face and
content validity and conduct a pretest of the instrument for reliability. Although assessment of
the validity and reliability of new surveys can be costly and time consuming, such testing is key
to identifying methodological flaws that could make the results suspect.
Face and content validity can be assessed by conducting cognitive interviews or convening one
or more focus groups that include targeted survey respondents or individuals with subject matter
expertise. A factor analysis could also be conducted to verify that the individual items that
comprise a scale are measuring the domain of interest. Reliability can be assessed using the testretest method in which the survey is administered to the same group at two different times. A
correlation coefficient is calculated and indicates the reproducibility of results. Correlation
coefficients with r-values at or above 0.70 indicate good reliability. However, even with high
reliability, a new survey instrument will have limited benchmarks for comparison of results.

Activity I.4: Develop the Sampling Plan
The EQRO should develop a sampling plan that represents
all eligible enrollees within the MCP (Worksheet 6.4). Refer
to Appendix B. Sampling Approaches for EQR Data
Collection Activities for an overview of sampling
approaches that can be used for drawing a survey sample. In
general, the sampling plan should incorporate information
from the five steps described below.

Resource for Activity 4

Step 1: Define the Study Population

Appendix B. Sampling Approaches for
EQR Data Collection Activities

WORKSHEET 6.4

Worksheet 6.4. Sampling Plan
• Provides a set of questions to assess
the sampling plan

• Provides an overview of sampling

The EQRO must first define the population to be studied (for
approaches and guidance for
example, all Medicaid or CHIP beneficiaries enrolled in an
determining sample sizes for EQR
data collection activities
MCP or all children with chronic conditions) and then
determine which data source(s) to use to construct a list of
all units in the study population (this list is referred to as the sampling frame). The sampling
frame will be used to draw the sample for data collection. The sampling frame should include all
the information necessary to determine whether units in the population are eligible for the study
276 | PROTOCOL SIX

(e.g., dates of Medicaid coverage and MCP enrollment) and any information that would be used
for stratification by subgroup (e.g., age, gender, zip code of residence).

Step 2: Determine the Type of Sampling to be Used
There are two basic types of sampling methods.
•

Probability (or random) sampling methods leave selection of population units to chance
and not to convenience or preference on the part of the individuals conducting the study or
otherwise participating in the study. Probability sampling removes systematic bias in the
selected sample due to observed and unobserved differences in the sampling units

•

Non-probability sampling methods are used when subjects are scarce or hard to sample (no
sampling frame) and/or the study relies on volunteers. The sample is based on the choice of
those administering the survey rather than chance; therefore, some bias can be expected

Probability sampling is preferable to non-probability sampling when feasible because it removes
systematic bias from the sample. For more information on commonly used types of probability
and non-probability sampling methods, see Appendix B.

Step 3: Determine the Number of Units to Sample
The number of units selected in the sample depends on several factors, including the level of
precision required to achieve statistically valid results, the expected number of respondents (i.e.,
the response rate), and other constraints on the financial and personnel resources available to
administer the survey. Samples with a larger number of units will provide a higher level of
precision, but may be more expensive to collect data from and present more of a burden on
financial and personnel resources.
For the CAHPS® Medicaid Health Plan Survey, the target number of completed surveys is 411
per health plan. The recommended initial sample sizes are 1,350 for the Medicaid Adult Version
and 1,650 for the Medicaid Child Version. For other surveys, the EQRO should consider
contacting a sampling statistician to conduct a statistical power analysis to help determine the
optimal number of units to sample to meet precision targets while accounting for financial and
personnel burden.
Given the interdependence between sample size, response rate, and precision, a goal is to achieve
the highest response rate possible. See Activity I.5 for strategies to maximize survey response.

Step 4: Select the Sample
In this step, the EQRO determines how the sample will be selected. For probability sampling
methods, the sample can be drawn using statistical software packages. For non-probability
samples, the sample is selected by the EQRO based on convenience or perceived
representativeness of the study population. The sampling plan should clearly explain the
PROTOCOL SIX | 277

sampling methods, and describe the procedures used to minimize bias. For more information on
selecting the sample, see Appendix B.

Activity I.5: Develop a Strategy to Maximize Response
The EQRO should develop a strategy for maximizing survey
response that includes a plan for both locating and contacting
the sample members.

Step 1: Maximize Completeness of Sample
Information Before Survey Launch
Before the survey is implemented, the EQRO should identify
the specific data it needs to locate sample members and
develop a strategy for ensuring the locating information is
complete. The following information is frequently used to
locate sample members in Medicaid surveys:
•

First and last name

•

Address

•

Home and cell phone numbers

•

E-mail address

•

Date of birth

•

Primary language

•

Preferred language

•

Name of MCP

•

Length of enrollment

WORKSHEET 6.5
Resource for Activity 5
Worksheet 6.5. Strategy to Maximize
Response
• Provides a set of questions to assess
the strategy for locating sample
members and specific data needed to
administer the survey

These data elements should be used for the purpose of contacting sample members and should be
kept separate from the survey data to protect the confidentiality of the sample members’ survey
responses and protected health information (PHI). The survey data provided by the sample
member should be identified by a unique, numeric identification number, not by name or other
identifying characteristic.
The EQRO should collect complete contact information and consider that some information may
be verified through the state’s eligibility files or the MCPs’ enrollee files. The EQRO may also
need to establish a data use agreement with the state or MCP for the protection and handling of
PHI. The EQRO should also expect missing data in the state data files and document its plans to
locate and contact respondents, including sending names in the sample file to a telephone
number look-up vendor or using a change-of-address database vendor.
278 | PROTOCOL SIX

Step 2: Design a Data Collection Strategy that Maximizes Response
The EQRO should design a data collection strategy that maximizes response and fits within the
available budget and schedule. The data collection strategies described below represent best
practices in the field of survey research and are frequently used to maximize survey response. 72
The EQRO should design a data collection plan that uses some or all of the strategies described
below:
1. Advance letter. Including an introductory letter before starting data collection lends
legitimacy to the survey. A good letter emphasizes survey sponsorship (e.g., on state
government letterhead signed by the agency director), describes the purpose of the survey,
includes a statement about sample member confidentiality, provides information on how the
sample member was selected for the survey, and describes benefits to the sample member as
a result of participation (and emphasizes there is no penalty for not responding). In addition,
the letter should be personalized and addressed to the respondent by name
2. Multiple and varied call attempts. Best practices to increase survey response include the
use of varied contact attempts. This can include multiple contact attempts at different times
of the day, mailing a reminder postcard or second survey if conducting a self-administered
paper survey, making a follow-up phone call to non-respondents to a mail survey, or
conducting repeat calls in a telephone survey. The EQRO should track and follow up on the
number of respondents that could not be contacted or failed to respond
3. Multi-mode surveys. Combining two or more modes of data collection (such as mail and
phone) in a single survey effort can lead to higher response rates than single-mode surveys.
This is because multi-mode surveys may:

Lower costs by beginning data collection in a cost-effective mode

Allow the data collection to continue for longer periods

Increase the timeliness of response (for example, results from a web-based survey can be
received faster than results by mail)

Limit coverage error by offering the survey by mail or web if the population of interest
may not have consistent telephone service. In addition, as more households have access
to the internet, using email and text messaging (with respondent permission) has become
an increasingly common method to contact respondents. Box 6.4 provides additional
information about integrating web-based outreach in data collection.

4. Multiple languages. The state and MCPs should have information about each beneficiary’s
primary and/or preferred language. If this information is not readily available, the EQRO

For more information about best practices see: (1) Dillman, Don A., Jolene D. Smyth, Leah Melani Christian. Internet, Mail
and Mixed-Mode Surveys: The Tailored Design Method (4th ed.), 2014. (2) Groves, R.M. Non response rates and
nonresponse bias in household surveys. Public Opinion Quarterly, vol. 70, no. 5, Special Issue 2006, pp. 646–675.

PROTOCOL SIX | 279

might include a sentence in the advance letter translated into the most common languages in
the area, inviting the individual to call for more information or to request a specific
translation
Box 6.4. Integrating Web-Based Outreach in Data Collection
• Send the advance letter by email in place of, or in addition to, first class mail
• Send the reminder postcard by email in place of, or in addition to first class mail
• Include a hyperlink to an online version of the survey in the advance letter or advance email
• Begin data collection with an online survey and follow-up with non-respondents by mail and/or telephone
• When deciding which mode(s) to use, always consider the population's access to and preferences for each mode

The strategies should be tailored to the survey population. In particular, the EQRO should
customize strategies for provider surveys. While the design and contact strategies listed above
also are effective when surveying health care professionals (such as physicians, nurses, and
practice staff), health care professionals historically are a difficult population to reach. A metaanalysis 73 of 154 surveys of health care professionals found statistically significant
improvements in response rates when using mail-based data collection (compared to the webbased mode) and when monetary incentives were offered (compared to those that did not offer a
monetary incentive or offered a non-monetary incentive). In addition, surveys with one or two
follow-up attempts yielded higher response rates than studies with three or more follow-ups.

Step 3: Specify the Method Used to Calculate the Response Rate
The sampling plan should specify the method that will be used to calculate the response rate. The
EQRO should use a standard methodology to calculate the response rate. The American
Association of Public Opinion Research (AAPOR) provides a list of standard definitions, which
is available at https://www-archive.aapor.org/AAPOR_Main/media/publications/StandardDefinitions20169theditionfinal.pdf.
The sampling plan should also note target response rates for similar surveys, which can be used
as a benchmark to assess the adequacy of the response rate after the survey is implemented, such
other surveys conducted by the state or by other states, or other types of surveys implementing
the same methodology.

Step 4: Include a Plan for a Non-Response Analysis
Finally, after the survey is complete, a non-response analysis should be conducted, as discussed
in Activity I.8. The sampling plan should describe the approach that will be taken to a non-

Young Ik Cho, Timothy P. Johnson, and Jonathan B. VanGeest. Enhancing Surveys of Health Care Professionals: A MetaAnalysis of Techniques to Improve Response. Evaluation & the Health Professions, vol. 36, no. 3, pp. 382-407, August 2013.

280 | PROTOCOL SIX

response analysis to assess whether there are differences between respondents and nonrespondents.

Activity I.6: Develop a Quality Assurance Plan
The EQRO should develop a quality assurance plan that
contains quality checks for all phases of the data
collection effort (Worksheet 6.6). The quality assurance
plan should describe the checks to be performed and the
processes used to implement the checks (see Box 6.5).
The quality checks should cover the sampling and locating
processes, be customized by data collection mode, and
specify data quality controls.

WORKSHEET 6.6
Resource for Activity 6
Worksheet 6.6. Quality Assurance Plan
• Provides a set of quality check
questions to assess the quality
assurance plan

Box 6.5. Tips for Quality Assurance Checks
The quality assurance plan should describe each quality check and clearly identify:
• What checks are being performed
• How the checks are performed
• Who performs the checks
• Frequency of the checks
• Percentage of survey records that are to be checked
• Corrective actions required if an issue is identified
• How the issue was resolved

Activity I.7: Implement the Survey According to the Work
Plan
The EQRO should implement the survey according to the
weekly data collection schedule laid out in the work plan
(see Activity I.2 and Worksheet 6.7). Although there is no
set time frame for data collection, on average, data
collection activities range from 10 to 14 weeks. Three
sample data collection schedules by week are included in
Table 6.2. Any deviations from the work plan should be
documented and the reasons for those deviations should
be explained.

WORKSHEET 6.7
Resource for Activity 7
Worksheet 6.7. Survey Implementation
According to the Work Plan
• Provides a set of questions to assess
survey implementation

PROTOCOL SIX | 281

Table 6.2. Sample Data Collection Schedules by Week
Data
Collection
Week

Mail-only protocol

Telephone-only Protocol

Mixed-mode Example:
Mail with Telephone Follow-up

Mail initial survey with cover
letter to sample members

Mail advance letters and begin
telephone contact to sample
members

Mail initial survey with cover
letter to sample members

Mail optional postcard and
receipt returned surveys

Mail optional postcard and
continue telephone follow-up

Mail optional postcard and
receipt returned surveys

Receipt returned surveys

Continue telephone follow-up
(weeks 3–10)

Receipt returned surveys

Mail second survey with cover
letter to non-respondents

Receipt returned surveys (weeks .
5–10)

Receipt returned surveys

Telephone follow up to nonrespondents and receipt returned
surveys (weeks 6–12)

End data collection

Activity I.8: Prepare and Analyze Survey Data and Present
Results in a Final Report
Once the surveys have been completed and returned, the
EQRO must prepare the data for analysis. This may
include post-processing procedures (e.g., cleaning and
editing, creating weights, and conducting a nonresponse
analysis). Then the EQRO proceeds with the data
analysis in accordance with the work plan and prepares
the final report (Worksheet 6.8).

WORKSHEET 6.8
Resource for Activity 8
Worksheet 6.8. Survey Data Analysis
and Final Report
• Provides a set of questions to assess
the data analysis and final report

Step 1: Implement Post-Processing
Procedures
Consistent with the quality assurance plan (Activity I.6), the EQRO should implement
procedures to handle responses that fail edit checks, address missing data, and remove data from
surveys determined to be unusable. The EQRO should specify the criteria used to remove
282 | PROTOCOL SIX

surveys or data from the final analytic file (including the threshold used to determine a
completed case). The EQRO should document the reasons for all exclusions or adjustments of
data used for the analysis.

Step 2: Calculate the Sampling Weights
When a sample is selected in such a way that there are different probabilities of selection for
different units, sampling weights must be constructed and used for any analyses conducted with
data collected from the sample. The weights take into account the sample design and
nonresponse of sampled units. The weighted results, therefore, are representative of the
population not just the units that responded to the survey.
The sampling weight is equal to one over the probability of selection for a unit. For example, if
the probability of selection is 0.25 for a unit, the sampling weight is 1/0.25 = 4. In probability
sampling, the sampling weights are used to make inferences to the study population. The
sampling weights would also need to be adjusted to account for nonresponse if there is
considerable nonresponse during data collection. The EQRO should consult a sampling
statistician to help calculate the sampling weights.

Step 3: Conduct a Non-Response Analysis
The response rate is only one indicator of survey quality. Another indicator is the extent to which
non-respondents may differ from respondents on the key variables in the survey sample. Because
significant differences may bias the survey estimates, it is important to conduct a nonresponse
analysis to assess the representativeness of the survey respondents.
Before beginning the data analysis, the EQRO should compare the characteristics of respondents
and non-respondents using means and frequency distributions. The analysis should rely on
information available in the sample frame (such as information found in state Medicaid
eligibility files). Tests of statistical significance (e.g., t-tests and chi-square tests) should be
performed to determine whether the differences are statistically significant. If there are
substantial differences between respondents and non-respondents, the EQRO should consult a
statistician to assess what types of adjustments might be necessary to account for potential bias
in the survey responses.

Step 4: Analyze Survey Data
Following the analysis plan laid out in the work plan and approved by the state, the EQRO
should generate means or frequency distributions for each survey question and calculate
statistics. The analysis should include a description of the population characteristics,
performance on the outcome measures included in the survey (such as access, timeliness, and
quality of care or experience of care).

PROTOCOL SIX | 283

In addition, the EQRO should examine differences in survey results among MCPs, between
MCPs and the FFS or PCCM population (if applicable), or between MCPs in the state and
nationally or regionally (if benchmarks are available). The EQRO could also analyze and report
on variations among subpopulations within each MCP. For example, the state may be interested
in whether responses differ significantly across geographic locations, racial/ethnic groups,
socioeconomic groups, or other identifiable subgroups. For recurring surveys with trendable
results, the EQRO could examine changes over time on key metrics.
Results should be weighted, account for the complex sample design in computing variances (if
applicable), and take into consideration the adequacy of sample sizes to support the analyses.
Some surveys include open-ended, qualitative responses related to experience or satisfaction. In
such cases, the open-ended responses should be reviewed, coded into categories if feasible, and
synthesized for analysis. Such information can enrich the quantitative data analysis and provide a
“voice” to illustrate the numerical findings.

Step 5: Prepare and Submit a Final Report
The EQRO should prepare and submit reports in the agreed format, which may include:
•

Survey purpose and objectives

•

Survey implementation procedures, including challenges encountered, lessons learned, and
recommendations for improving future efforts

•

Overview of analytic findings, including subgroup analyses and tests of statistical
significance

•

Methodologically appropriate, comparative information about MCP performance

•

A detailed assessment of each MCP’s strengths and weaknesses with respect to access,
quality, and/or timeliness of health care furnished to enrollees

•

Conclusions drawn from the data

Results from the survey should always be presented for groups and not for individual
respondents. Statistical graphs should accompany narrative text to aid comparison and
interpretation. For example, bar graphs and comparison charts, such as those recommended by
CAHPS®, convey important information about the performance of each MCP and indicate
meaningful differences among MCPs.
The EQRO should submit a draft report and provide the state with an opportunity to review and
comment on the draft report. The EQRO should then revise the draft and submit a final report
that incorporates state comments. Other deliverables may include a raw data file and analysis
files, as well as public reports, presentations, or web sites developed for public reporting.

284 | PROTOCOL SIX

Section II. Validating a Survey
Protocol 6 also contains guidance for EQROs charged with validating a survey conducted by a
state, MCP, or a vendor hired by the state or MCP. 74 The activities described in this section
focus on reviewing the survey design and implementation for validity, reliability, and
methodological rigor. They do not include collecting additional survey data from survey
respondents to verify their responses or test for survey validity and reliability.
The EQRO should use the Worksheets for Protocol 6 or a similar tool to guide the validation
process. The EQRO should identify the documentation it used to review the survey procedures
and note its findings for each activity. In addition, the EQRO should note the absence of
documentation for a particular activity as it may be relevant to the survey validation.
Upon completion of the validation activities, the EQRO should synthesize all of the validation
findings from Activities II.1 through II.8 based on the findings documented in the Worksheets
for Protocol 6. The EQRO should submit a final validation report that assesses the overall quality
of the survey, and in particular, the extent to which the survey achieved its purpose and
objectives. Key elements of this assessment are whether the survey findings can be generalized
to the population from which the sample was drawn and whether the data quality and
completeness can support the survey’s intended uses.
Although survey validation is an optional EQR-related activity, CMS recommends that surveys
be validated when states intend to use survey results for such decisions as consumer health plan
selection, health plan or provider payment, or performance incentives (e.g., auto-assignment).

Activity II.1: Review the Survey Purpose, Objectives, and
Audience
To understand and evaluate the adequacy of the survey to
meet its intended uses, the EQRO should seek
information from written sources or through interviews
about the survey’s purpose, objectives, and audience. See
Activity I.1 for more information about defining the
survey purpose, objectives, and audience.

WORKSHEET 6.1
Resource for Activity 1
Worksheet 6.1. Survey Purpose,
Objectives, and Audience
• Provides a set of questions to assess
the clarity of the survey purpose,
objectives, and audience

Many states and MCPs contract with survey vendors certified by the National Committee for Quality Assurance (NCQA) to
conduct CAHPS® 5.0H surveys following a standardized and validated protocol. A list of approved vendors is available at
https://www.ncqa.org/programs/data-and-information-technology/hit-and-data-certification/cahps-5-1h-surveycertification/vendor-directory/.

PROTOCOL SIX | 285

Activity II.2: Review the Work Plan
To understand the survey implementation plan, the EQRO
should review the work plan, including the project
management plan, schedule, reporting requirements, data
preparation plan, data analysis plan, and security protocols
and procedures. The work plan provides a foundation for
understanding the rigor of the overall survey approach;
deviations from the work plan may signal concerns related to
the effectiveness of survey implementation. See Activity I.2
for more information about developing a work plan.

WORKSHEET 6.2
Resource for Activity 2
Worksheet 6.2. Work Plan
• Provides a set of questions to assess
the work plan

Activity II.3: Review the Reliability and Validity of the Survey
Instrument
As discussed in Activity I.3, there are three options for
selecting a survey instrument:

WORKSHEET 6.3

1. Use an existing validated survey instrument

Resource for Activity 2

2. Adapt an existing survey instrument with additional statespecific questions

Worksheet 6.3. Survey Instrument

3. Develop a new survey instrument

• Provides a set of questions to assess
the selection of the survey instrument

Each of these approaches involves trade-offs. For example, use of an existing validated survey
instrument increases assurances about the instrument’s validity and reliability, but the instrument
may have gaps in survey content for the specific survey purpose. Development of a new survey
instrument may result in more targeted content for the specific survey purpose, but require more
effort to ensure validity and reliability of the instrument. As part of this validation activity, the
EQRO is charged with assessing the extent to which there is sufficient documentation of the
validity and reliability of the selected survey instrument.
The EQRO should not conduct independent validity and reliability testing of the survey
instrument; however, it should note whether such testing was done. The EQRO should consider
the adequacy of the survey’s reliability and validity testing in determining whether to rely on the
survey findings to inform the EQRO’s analysis and evaluation of access, quality, and timeliness
of health care. See Activity I.3 for more information about assessing the validity and reliability
of survey instruments.

286 | PROTOCOL SIX

Activity II.4: Review the Sampling Plan
The EQRO should assess the sample plan documentation
for the following:
1. Clear definition of the study population. The EQRO
should document whether there was a clear definition of
the study population
2. Appropriate specifications for the sample frame. The
EQRO should assess whether the sampling frame was
clearly described and appropriate to the survey
objectives

WORKSHEET 6.4
Resource for Activity 4
Worksheet 6.4. Sampling Plan
• Provides a set of questions to assess
the sampling plan
Appendix B. Sampling Approaches for
EQR Data Collection Activities
• Provides an overview of sampling
approaches and guidance for
determining sample sizes for EQR
data collection activities

3. Quality of the sampling frame. The EQRO should
assess whether the sampling frame is free from bias.
The sampling frame should include all members of the
population to be studied, and not omit any members of the population

4. Type of sampling method used. The EQRO should evaluate whether the sampling method
used was appropriate to the survey’s purpose (e.g., use of probability versus non-probability
methods). For more information, see Appendix B
5. Adequacy of the sample size. The EQRO should determine whether the sample size was
appropriate for the survey. Two factors influence the determination of the appropriate sample
size for a survey: (1) the acceptable margin of error, and (2) the confidence levels
6. Procedures for sample selection. The EQRO should review the sample selection procedures
including reviewing the statistical program or other process used to generate the sample. The
EQRO should determine the extent to which the selection of sample members was conducted
to protect against bias
The level of detail involved in this review requires that the EQRO use professional statisticians.
The EQRO must evaluate whether the sample selected was sufficiently representative of the
study population for the EQRO to have confidence in the survey findings. See Activity I.4 for
more information about developing a sampling plan.

PROTOCOL SIX | 287

Activity II.5: Review the Adequacy of the Response Rate
In this activity, the EQRO should review the methods
used to maximize the response rate, as well as the
methods used to calculate the response rate. In addition,
the EQRO should assess potential sources of nonresponse and bias, and the extent to which the response
rate weakens or strengthens the generalizability of the
survey findings.

The EQRO should determine whether a standard
methodology was used to calculate the response rate. The
American Association of Public Opinion Research
(AAPOR) provides a list of standard definitions and response rate calculators on its website at
http://www.aapor.org/Standards-Ethics/Standard-Definitions-(1).aspx. To provide context for the
assessment of the adequacy of the response rate, the EQRO should consider benchmarking the
response rate against those achieved by similar surveys. As discussed in Activity I.8, a
nonresponse analysis can provide insights into the representativeness of the survey when
response rates are low.
See Activity I.5 for more information on strategies to maximize response.

Activity II.6: Review the Quality Assurance Plan
The EQRO should review the quality assurance plan to
ensure that it contains quality checks for all phases of the
data collection effort. Specific areas for focus include
checks during the sampling and locating processes,
customization by data collection mode, and specification of
data quality controls. In addition, the EQRO should be sure
that the plan specifies how the checks will be implemented.
See Activity I.6 for more information on the quality
assurance plan.

288 | PROTOCOL SIX

WORKSHEET 6.6
Resource for Activity 6
Worksheet 6.6. Quality Assurance Plan
• Provides a set of quality check
questions to assess the quality
assurance plan

Activity II.7: Review the Survey Implementation
The EQRO should review documentation regarding the
survey implementation and assess whether implementation
conformed to the work plan. The EQRO should specifically
consider the following:

WORKSHEET 6.7
Resource for Activity 7
Worksheet 6.7. Survey Implementation
According to the Work Plan

•

Adherence to the sampling plan

•

How the survey questionnaire was administered,
including formatting and distribution of mailed surveys
or scripting and training of telephone interviewers

•

Changes to the survey schedule

•

Evidence of implementation of the quality assurance checks

•

Problems detected and corrections implemented during the survey process

•

Confidentiality procedures followed

•

Data collection, data entry, and data quality control methods used, including reports of
missing data, data that failed edit checks, and incomplete or unusable surveys

• Provides a set of questions to assess
survey implementation

See Activity I.7 for more information on survey implementation.

Activity II.8: Review the Survey Data Analysis and Final
Report
The EQRO should review how the survey data were
analyzed, including the statistical procedures used and
comparisons made. The EQRO should assess whether the
analysis was appropriate to the survey purpose, whether
appropriate statistical tests were applied, and how well the
survey findings were supported by the data. In its final
validation report, the EQRO should document its
conclusions and provide written findings on:

WORKSHEET 6.8
Resource for Activity 8
Worksheet 6.8. Survey Data Analysis
and Final Report
• Provides a set of questions to assess
the data analysis and final report

•

The survey’s technical strengths and weaknesses

•

Appropriateness of analysis methods (e.g., data quality, sample sizes, weighting and
adjustment for complex sample design if applicable, significance testing)

•

Appropriateness of presentation approaches (such as text, tables, figures)

•

Appropriateness of conclusions drawn from the survey data

•

The limitations and generalizability of survey findings

PROTOCOL SIX | 289

See Activity I.8 for more information about the presentation of survey findings.

Technical Appendix for Protocol 6: Understanding Potential
Sources of Survey Error
Survey results are used increasingly for “high-stakes” activities such as consumer health plan
selection, health plan or provider payment, or performance incentives (e.g., auto-assignment). As
a result, there is increasing scrutiny on the quality and integrity of surveys to support such
initiatives. States and MCPs cannot afford “errors” in a survey, as the consequences may be
substantial from a beneficiary, provider, health plan, and state perspective.
This appendix provides additional information on how to assess the overall quality of the survey
effort using a Total Survey Error (TSE) paradigm. The TSE paradigm identifies potential sources
of survey error and examines the accumulation of all errors that arise in the design, collection,
processing, and analysis of survey data. It is important to note that in the TSE paradigm, errors
are sources of uncertainty, a deviation of a survey response from its underlying true value. Errors
are not mistakes.
The TSE paradigm is included in Figure 6.2. Table 6.3 describes which errors may arise in the
process and steps that EQROs (or survey vendors) can take to remedy the errors. This
information can inform the administration and validation of surveys to improve overall survey
quality.

290 | PROTOCOL SIX

Figure 6.2. The Total Survey Error Paradigm: Understanding Potential Sources of Error in
Surveys

Source: Adapted from Groves, R.M. et al, Survey Methodology, Hoboken, NJ, John Wiley & Sons, 2004.

Table 6.3. Mapping Common Sources of Survey Error to Data Collection Activities
and Remedies to Minimize Error
Activity

Common
Errors

Definition

Remedies

Specification
error

Sometimes called validity, are we
measuring what we say we are
measuring?

• Use validated scales, pretesting, cognitive
testing, focus groups

Measurement
error

When an answer to a question is
inaccurate, imprecise, or cannot be
compared to other respondents’
answers due to the questionnaire,
respondent, interviewer, or mode

Design survey using best practices:
• Programming checks
• Validated scales
• Multiple languages
• Multiple modes to participate

Coverage error When who/the group you want to study • Offer multiple modes to participate
differs from who is available to study
• Use dual frame samples

PROTOCOL SIX | 291

Activity

Common
Errors

Sampling error

Nonresponse
error

Definition

Remedies

When the survey includes only a
subset of the target population. This
error cannot be avoided unless a
census is conducted

• Conduct power calculations

When people in the survey sample do
not respond and are different from
those who do respond in a way that is
important to the study. There is Unit
nonresponse (sample members who
do not respond to the survey) and Item
nonresponse (sample members who
skip or refuse specific questions)

• Use proven contact strategies, such as
advance letters and vary modes for
nonresponse follow up

• Create sampling weights
• Conduct all analyses using weights

• Institute range checks
• Monitor skip patterns and missing data
through frequency reviews
• Conduct critical item retrieval
• Conduct nonresponse bias analysis
• Make nonresponse adjustments

Processing
error

Problems that occur when preparing
• Develop cleaning and coding specifications
“raw” datasets set for analysis, such as • Perform double entry, adjudication, data
inconsistent coding, treatment of
review
outliers, or deriving new variables

Adjustment
error

Mistakes in efforts to improve the
• Use post-survey adjustments such as
quality of the survey estimates as a
weighting and imputation
result of coverage, sampling, and nonresponse errors

Inferential error Making opinionated statements,
• Prepare comprehensive quality assurance
drawing incorrect conclusions, or going
plans
beyond the limits of the design
• Develop rigorous data analysis plans

Source: Adapted from Groves, R.M. et al, Survey Methodology, Hoboken, NJ, John Wiley & Sons, 2004.

END OF PROTOCOL 6

292 | PROTOCOL SIX

Worksheets for Protocol 6: Survey Administration and
Validation Tools
Instructions. Use these or similar worksheets as a guide when administering or validating a survey. Each numbered
worksheet corresponds to an activity in the protocol. For each question, please check “Yes,” “No,” or “Not applicable.”
If the answer is “No” or “Not applicable,” please explain in the “Comments” column. Add “Comments” for any question
as needed.
•

For survey administration: Use the worksheets to track and document steps performed in designing and
implementing the survey. In the “Comments” column, document decisions or findings

•

For survey validation: Use the worksheets to track and document steps performed in validating the survey. In the
“Comments” column, document the outcome of validation activities, including sources reviewed. The worksheets
can also be used as an outline for the final report to the state. Expand the tool to include other activities or
findings as needed

This tool includes the following worksheets and the applicable activity and step:
Worksheet Name

Protocol Activity and Step

Section I. Administering the Survey
Section II. Validating the Survey
Worksheet 6.1. Survey Purpose,
Objectives, and Audience

Section I. Activity 1. Define the Survey Purpose, Objectives, and
Audience
Section II. Activity 1. Review the Survey Purpose, Objectives, and
Audience

Worksheet 6.2. Work Plan

Section I. Activity 2. Develop the Work Plan
Section II. Activity 2. Review the Work Plan

Worksheet 6.3. Survey Instrument

Section I. Activity 3. Select the Survey Instrument
Section II. Activity 3. Review the Validity and Reliability of the Survey
Instrument

Worksheet 6.4. Sampling Plan

Section I. Activity 4. Develop the Sampling Plan
Section II. Activity 4. Review the Sampling Plan

Worksheet 6.5 Strategy to Maximize
Response

Section I. Activity 5. Develop a Strategy to Maximize Response

Worksheet 6.6. Quality Assurance Plan

Section I. Activity 6. Develop a Quality Assurance Plan

Section II. Activity 5. Review the Adequacy of the Response Rate
Section II. Activity 6. Review the Quality Assurance Plan

Worksheet 6.7. Survey Implementation
According to the Work Plan

Section I. Activity 7. Implement the Survey According to the Work Plan

Worksheet 6.8. Survey Data Analysis
and Final Report

Section I. Activity 8. Prepare and Analyze Survey Data and Present
Results in a Final Report

Section II. Activity 7. Review the Survey Implementation

Section II. Activity 8. Review the Survey Data Analysis and Final Report

PROTOCOL SIX | 293

Worksheet 6.1. Survey Purpose, Objectives, and Audience
Survey purpose, objectives, and audience: ________________________________________________________
Assess the clarity of the survey purpose and audience by answering the following questions. Insert comments to
explain “No” and “Not applicable” responses.

Question

Yes

Not
applicable

Comments

Was there a clear, written statement of the survey
purpose that addresses access, timeliness, and/or
quality of care?
Was the unit of analysis clearly stated?
Did the unit of analysis include individual MCPs?
Was there a clear and measurable written study
objective?
Was the audience for and intended use of the survey
findings identified?
Overall validation assessment: In the comments
section, note any recommendations for improving the
survey purpose, objective, and audience
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

294 | PROTOCOL SIX

Worksheet 6.2. Work Plan
Date of work plan: ______________________________________________________
Assess the adequacy of the work plan by answering the following questions. Insert comments to explain “No” and
“Not applicable” responses. (Note: Validation of the work plan occurs in conjunction with Activity 5, Review Survey
Implementation According to the Work Plan.)

Question

Yes

Not
applicable

Comments

Did the work plan include a project management plan
(including key staff and roles)?
Did the work plan include a project schedule
(including timelines and deliverable dates)?
Did the work plan specify project reporting
requirements (including the number, format, and
content of the reports)?
• The work plan should include a description of any
reports that the EQRO will be responsible to
publicly release, if this is part of the EQRO’s
scope of work
Did the work plan include a data preparation plan,
such as production of data files, data file format, and
delivery?
Did the work plan include a data analysis plan
(including the use of a statistician as appropriate)?
• The EQRO should use a statistician to develop an
analysis plan that supports the survey purpose
and objectives and is consistent with the intended
use of results
• If feasible, the EQRO should provide the state with
a mock-up of the analysis before administering the
survey. This will assure the survey analysis will be
consistent with the intended use of results
Did the work plan include data security protocols and
procedures for assuring the confidentiality of data in
compliance with HIPAA?
Overall validation assessment: In the comments
section, note any recommendations for improving the
work plan
Acronyms: CHIP = Children’s Health Insurance Program; EQR = External Quality Review; EQRO = External Quality Review
Organization; IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care
Organization; MCP= Managed Care Plan; PAHP= Prepaid Ambulatory Health Plan; PCCM = Primary Care Case Management;
PIHP = Prepaid Inpatient Health Plan; PIP = Performance Improvement Project.

PROTOCOL SIX | 295

Worksheet 6.3. Survey Instrument
Name of survey instrument______________________________________________________________________
Assess the selection of the survey instrument by answering the following questions. Insert comments to explain “No”
and “Not applicable” responses. Complete a separate worksheet for each survey instrument.

Question
Was the selected survey instrument appropriate for
the purpose of the survey and the unit of analysis?
Were new items developed for the survey?
If new items were developed, was a test of validity
and reliability conducted for the new items?
Was the overall survey instrument tested for face
validity and content validity and found to be valid?
Was the overall survey instrument tested for
reliability and found to be reliable?
Was testing performed for the specific target
population (e.g., Medicaid or CHIP) and languages?
Overall validation assessment: In the comments
section, note any recommendations for improving the
selection of the survey instrument

296 | PROTOCOL SIX

Yes

Not
applicable

Comments

Worksheet 6.4. Sampling Plan
Assess the sampling plan by answering the following questions. Insert comments to explain “No” and “Not applicable”
responses.

Question

Yes

Not
applicable

Comments

Was the study population clearly defined?
Was the sampling frame clearly defined and
appropriate based on the survey objectives?
Was the sampling frame free from bias?
Was the sampling method appropriate to the survey
purpose?
Was the sample size sufficient for the intended use of
the survey (acceptable margin of error, level of
certainty required)?
Were the procedures used to select the sample
appropriate and protected against bias?
Overall validation assessment: In the comments
section, note any recommendations for improving the
sampling plan

PROTOCOL SIX | 297

Worksheet 6.5. Strategy to Maximize Response
Assess the strategy for locating sample members and specific data needed to administer the survey by answering the
following questions. Insert comments to explain “No” and “Not applicable” responses.

Question
Was locating of sample members conducted to
ensure complete contact information?
Locating is a technique used to improve response
rates by locating and contacting sample members.
This includes verified collection of data, such as first
and last name, home address, email address, phone
number(s), date of birth, language preference, etc.
Were any of the following strategies included to
maximize response:
• Advance letter
• Multiple and varied call attempts
• Multi-mode surveys
• Multiple languages
Were strategies customized to the study population
(e.g., providers versus beneficiaries)?
Was the method specified for calculating the
response rate, and if so, was the method in
accordance with industry standards?
Was a plan included to conduct a non-response
analysis?
Overall validation assessment: In the comments
section, note any recommendations for improving the
response strategy

298 | PROTOCOL SIX

Yes

Not
applicable

Comments

Worksheet 6.6. Quality Assurance Plan
Date of Quality Assurance Plan: _________________________________________________
Assess the quality assurance plan by indicating whether the following quality checks were included in the plan. Insert
comments to explain “No” and “Not applicable” responses. (Note: The assessment of whether the plan was
implemented appropriately is included in Worksheet 6.7.)

Question

Yes

Not
applicable

Comments

Sampling. Did the plan include a check to ensure
the sample was constructed as specified in the
sampling plan?
Locating. Did the plan include a check that initial
contact was made for every sample member?
Mail data collection. Were the following quality
checks included in the plan?
• Was the survey reviewed for respondent reading
level (surveys should be written at a 6th grade
reading level to ensure most respondents are able
to read and understand the content)
• Were specifications and procedures developed for
formatting, reproducing, and distributing the
survey questionnaire?
• Were contents of the mailing packet, such as the
cover letter and questionnaire, reviewed for
accuracy, print smearing, fading, and
misalignment?
• Were the returned mail surveys data entry
reviewed for accuracy?
Telephone data collection. Were the following
quality checks included in the plan?
• Were interviewer training and telephone scripts
reviewed for accuracy?
• Were telephone interviews monitored to confirm
that interviewers read questions verbatim and
accurately captured responses?
Web-based data collection. Did the plan include a
check that the web-based instrument programming
and content was tested for accuracy?
Data quality controls. Did the plan include
procedures to handle responses that fail edit checks,
treatment of missing data, and determination of
usable/complete surveys? (Note: The plan should
establish a pre-determined number of questions that
must be answered by the respondent to be
considered a usable case.)
Overall validation assessment: In the comments
section, note any recommendations for improving the
quality assurance plan

PROTOCOL SIX | 299

Worksheet 6.7. Survey Implementation According to the Work Plan
Assess the implementation of the survey by answering the following questions. Insert comments to explain “No” and
“Not applicable” responses.

Question
Was the weekly data collection plan implemented as
described in the work plan?
If deviations from the data collection plan occurred,
were the reasons for the deviations explained?
Were quality assurance checks implemented as
specified in the quality assurance plan (see Worksheet
6.6)? If deviations occurred, please explain in the
Comments column
• Was the sampling plan verified to ensure the
sample was constructed as specified?
• Was initial contact made for every sample
member?
• Were specified quality checks made in accordance
with the data collection mode (mail, telephone,
web-based, or mixed mode)?
• Were procedures developed to handle responses
that fail edit checks, treatment of missing data, and
removal of surveys or data determined to be
unusable?
Overall validation assessment: In the comments
section, note any recommendations for improving the
implementation of the survey

300 | PROTOCOL SIX

Yes

Not
applicable

Comments

Worksheet 6.8. Survey Data Analysis and Final Report
Assess the data analysis and final report by answering the following questions. Insert comments to explain “No” and
“Not applicable” responses.

Question

Yes

Not
applicable

Comments

Were post-processing procedures implemented to
address the following:
• Responses that failed edit checks
• Missing data
• Removal of surveys or data determined to be
unusable
Were weights created as appropriate for analyzing
survey responses and generalizing results to the
study population?
Was a nonresponse analysis conducted to determine
if survey respondents differ from respondents on key
variables important to the findings?
Were survey data analyzed following the analysis
plan laid out in the work plan?
Did the final report include a comprehensive
overview of survey purpose/objective,
implementation, and substantive findings?
Overall validation assessment: In the comments
section, note any recommendations for improving the
data analysis and final report

END OF WORKSHEETS FOR PROTOCOL 6

PROTOCOL SIX | 301

Protocol 7. Calculation of Additional
Performance Measures
AN OPTIONAL EQR-RELATED ACTIVITY

ACTIVITY 2: CALCULATE MEASURES
ACTIVITY 3: REPORT RESULTS

Background
One purpose of quality measurement is to evaluate the degree to which
evidence-based treatment guidelines are followed, where indicated, and to
assess the results of care. The use of quality measurement helps strengthen
accountability and support performance improvement initiatives at numerous
levels. Performance measures can be used to demonstrate a variety of activities
and health care outcomes for particular populations. For example, states use
performance measures to monitor the performance of individual managed care
plans (MCPs) at a point in time, to track their performance over time, to
compare performance among MCPs, and to inform the selection and evaluation
of quality improvement activities.
Federal regulations at 42 CFR 438.330(c) require states to specify standard
performance measures for MCPs to include in their comprehensive quality
assessment and performance improvement (QAPI) programs. 75 Each year, the
MCPs must: (1) measure and report to the state standard performance measures
specified by the state; (2) submit specified data to the state which enables the
state to calculate the standard performance measures; or (3) a combination of
these approaches. Validation of the performance measures specified by the state
for inclusion in MCPs’ QAPI programs is a mandatory external quality review
(EQR)-related activity (see 42 CFR 438.358(b)(1)(ii)), as described in Protocol
2. Validation of Performance Measures Reported by the Managed Care Plan.
Federal regulations at 42 CFR 438.358(c)(3) specify that the external quality
review organization (EQRO) may calculate performance measures in addition to

PROTOCOL SEVEN

ACTIVITY 1: PREPARE FOR MEASUREMENT

More information about QAPI and performance measures is available at 42 CFR 438.330(b)(2). CHIP
regulations at 42 C.RF.R 457.1240(b).cross-reference QAPI and performance measure regulations.

302

those specified by the state for inclusion in MCPs’ QAPI programs. Calculation of these
additional performance measures are an optional EQR-related activity.
In many cases, states and MCPs use measures included in the CMS Child and Adult Core Set
measures to monitor and track quality of care in Medicaid and CHIP. 76 While use of these
measures by states is voluntary, CMS encourages states to adopt and use the Child and Adult
Core Set measures to support their managed care quality measurement and improvement
initiatives. Many Core Set measures are part of the Healthcare Effectiveness Data and
Information Set (HEDIS®), and have national and regional benchmarks.
This protocol provides guidance to states on the calculation of additional (non-QAPI)
performance measures to monitor the care provided by MCPs to enrollees covered by Medicaid
and CHIP.

Getting Started on Protocol 7
Protocol 7 consists of three activities: preparation for measurement, calculation, and reporting
(Figure 7.1). For each activity, the protocol specifies the steps to be performed and the outcomes
to be achieved. The remainder of this protocol outlines the steps associated with these activities.
Figure 7.1. Protocol 7 Activities

PROTOCOL SEVEN | 303

Two supplemental resources are available to help EQROs calculate additional performance
measures:
•

Worksheets for Protocol 7. Performance Measure Calculation Tools, which can be used to
identify the performance measures to be calculated, document the technical specifications for
each measure, develop a master list of data elements, indicate the data sources and any
known data quality issues, and specify the file format for the transmission of the required
data elements

•

Appendix A. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems

Activity 1: Prepare for Measurement
Step 1: Identify the Performance Measures to be Calculated
The state should provide the EQRO with a list of
performance measures to be calculated along with
technical specifications for their calculation. The
EQRO must understand the state’s specifications for
each performance measure (e.g., sampling guidelines,
data sources, measurement period, instructions for
calculating numerators and denominators), as well as
the state’s requirements for benchmarking, analysis,
and reporting.

WORKSHEET 7.1
WORKSHEET 7.2
Resources for Activity 1, Step 1
Worksheet 7.1. List of Performance
Measures to be Calculated
• Provides a template for identifying the
measures the EQRO will calculate for
the state, including the source, how
frequently to calculate each measure,
and when each measure is due to the
state

The EQRO must also understand the state’s
requirements for the timing and format of the
performance measure report. The EQRO should
create a list of performance measures to be calculated
Worksheet 7.2. Companion
Performance Measurement Tool
to document the measures required by the state and
the reporting frequency and timeline for each
• Provides a template for documenting
additional information about
measure (Worksheet 7.1). For each performance
measures in Worksheet 7.1, including
measure listed in Worksheet 7.1, the EQRO should
technical specifications, benchmarks,
complete a companion performance measurement
performance standards, or other
information about state requirements
worksheet that contains the technical specifications
for the measure, benchmarks, performance standards,
and other information needed to analyze the performance measure according to the state’s
304 | PROTOCOL SEVEN

requirements (Worksheet 7.2). The EQRO may need to request clarification from the state,
measure steward, or other expert if the measure specifications are unclear.

Step 2: Prepare for Data Collection
The EQRO should send an introductory communication to the MCP outlining the purpose,
process, and timeline for its performance measure calculation activities. In addition, the EQRO
should request a contact within the MCP to schedule activities and provide requested documents
and other information.
The EQRO should inform the MCP that it may be necessary to interview MCP or vendor staff
with responsibility for data collection or performance measurement. The information provided
by the MCP should inform the EQRO of the location of the required data, which organization
(state, EQRO, or MCP) will need to collect and integrate specific data elements, and how to
access the data. Information obtained from MCP staff may improve the efficiency and accuracy
of the EQRO’s effort to collect and integrate the data necessary for calculating performance
measures.
During this step, the EQRO should also review or conduct an Information Systems Capabilities
Assessment (ISCA) for each MCP to:
•

Understand data sources, flows, and integration processes used by the MCP

•

Identify where the EQRO needs to work with outside data sources to obtain additional data

•

Determine which data elements are integrated by the MCP and which data elements the
EQRO must integrate

Appendix A contains the ICSA tool and instructions for completing the ISCA. For more
information on how the EQRO should conduct or review an existing ICSA as part of its
performance measurement activities, please refer to Protocol 2, Activity 1, Step 2, Assess the
Integrity of the MCP’s Information System.
If data will be collected from other sources such as state public health registries, vital records,
hospital discharge abstract databases, or behavioral health vendors under contract to the state, the
EQRO should establish contact with the organizations responsible for these data sources.

PROTOCOL SEVEN | 305

Step 3: Identify Required Data Elements, Data Sources, and Data Quality
Issues
Next, the EQRO should prepare a master list of data
elements (Worksheet 7.3) and identify available data
sources for each required data element, noting any
completeness or integration issues for each element
(Worksheet 7.4).

WORKSHEET 7.3
WORKSHEET 7.4
Resources for Activity 1, Step 3

Data sources may include those maintained by an MCP in
Worksheet 7.3. Data Element Master
a data repository, such as claims or enrollment data. Data
Checklist
sources may also include sources external to the MCP,
• Provides a template for identifying the
such as a state registry, provider medical record, MCP
data elements needed to calculate
each performance measure
vendor, or state vendor. The EQRO should document data
Worksheet 7.4. Data Availability and
capture or integration issues for the required data
Data Quality
elements, such as an inability to capture individual
• Provides a template for documenting
prenatal care services when the MCP pays for maternity
data availability and data quality
care using a global fee. As another example, the EQRO
issues (e.g., completeness and
may identify issues associated with data sources external
integration)
to the MCP, such as difficulty accessing confidential
information about mental health services that the state contracts with another organization to
manage, incomplete data in a voluntary state registry, or challenges in obtaining vital records
required for data linkage.

Activity 2: Calculate Measures
Step 1: Collect Performance Measure Data
After the required data elements and data sources have
been identified, the EQRO will request the data needed
to calculate the performance measures from the MCP or
other data suppliers. For each data source, the EQRO
should specify how the data are to be transmitted to the
EQRO, including appropriate privacy and security
safeguards. To ensure accurate and complete data for
measure calculation, the EQRO should develop a file
format that specifies the content and structure of the data
file along with definitions of all data fields (Worksheet 7.5).

WORKSHEET 7.5
Resource for Activity 2, Step 1
Worksheet 7.5. Illustrative File Format
for Transmission of Claims Data
• Provides a template for constructing
an electronic data shell or file format
including definitions for all data fields

The EQRO should construct file formats that are customized to each data supplier. The file
format for obtaining data from the MCP data repository will likely include all data elements that
originate from claims/encounter, eligibility, and provider transaction systems. (In some cases, the
data will be available in a state data repository and the file format should reflect the state system
306 | PROTOCOL SEVEN

structure.) A file format used to obtain vital records or immunization registry data would contain
different data fields and definitions applicable to those sources.
If the EQRO needs to conduct medical record review, it should develop the following resources:
•

Abstraction tools

•

Training for personnel conducting the medical record abstraction

•

Quality assurance procedures to assess the accuracy and reliability of the medical record
abstraction

•

Electronic data entry edits for abstracted medical record information

If the MCP or other entity is performing medical record review and supplying those data to the
EQRO, the EQRO should refer to Protocol 2 to validate the abstracted medical record
information.

Step 2: Clean Data
As the EQRO receives data, it should evaluate each incoming data stream to ensure that the
number of bits received is equal to the number sent. After entering the data into its repository,
the EQRO should clean the data using electronic edits. Examples of edits include the following:
•

Valid procedure codes (e.g., active code, required number of digits)

•

Valid diagnosis codes (e.g., active code, required number of digits)

•

Internal consistency of diagnosis and procedure codes (e.g., consistent with the enrollee’s age
or gender, or the practitioner’s specialty)

•

Correct field size and type (e.g., alpha, numeric, date)

•

Valid date ranges (e.g., “to” date is later than “from” date; dates occur during the appropriate
timeframe for the measure)

•

Valid practitioners (e.g., active provider)

•

Valid enrollees (e.g., eligible on date of service)

Data that pass the edit should be integrated in the EQRO’s performance measure repository (see
Step 3). When data fail an edit, the EQRO should contact the supplier and request the data be
corrected and resubmitted. The EQRO should document the nature and extent of failures,
including information about whether it received corrected information. This documentation is
necessary for the EQRO to understand the accuracy and completeness of the data underlying the
performance measures it will calculate.

PROTOCOL SEVEN | 307

Step 3: Integrate Data into Performance Measure Repository
The EQRO may receive data from multiple MCPs, multiple sources within each MCP, and other
organizations, such as a statewide registry or other state vendors responsible for delivering
specific benefits like pharmacy or mental health services. To calculate performance measures,
the data must be integrated across the various data sources so that all services provided to a
specific enrollee can be associated with that enrollee. The ISCA, reviewed in Activity 1, Step 3,
will provide information about the adequacy of data integration within each MCP.
During this step, the EQRO will also assess the integration of data from non-MCP data suppliers,
such as the state’s encounter data repository or other vendors. This may include administering
relevant portions of the ISCA to these other suppliers. The EQRO must determine which
portions of the ISCA are relevant depending on the specific data elements the supplier provides
and the degree of data integration the supplier must perform. The assessment includes assessing
the reliability of data transmissions within and from each data supplier. The EQRO may have
different degrees of access to these data suppliers and must work with them, to the extent
possible, to understand the data flows and procedures used to ensure data integrity. For each data
supplier, the EQRO may need to:
•

Examine the details of the data supplier’s processes to accurately and completely transfer
data from the transaction files (i.e., enrollment, provider, encounter/claims) into its data
repository, if any

•

Examine samples of data to assess completeness and accuracy

•

Investigate the data supplier’s processes to consolidate multiple files (sometimes referred to
as deduplicating or “de-duping” of files), and to extract required information from its data
repository

•

Compare actual results of file consolidations or extracts to those that should have resulted
according to documented algorithms or specifications

•

Review procedures for consolidating data from vendors in ways that ensure the accurate,
timely, and complete integration of the data

•

Review computer program reports or documentation that reflect these vendor coordination
activities, and spot check to verify that no data necessary to performance measure reporting
are lost or inappropriately modified during transfer

•

Assess the extent to which proper linkage mechanisms have been employed to join data from
all necessary sources (e.g., identifying an enrollee with a given disease/condition)

To ensure proper data integration within its own data repository, the EQRO must undertake the
following activities:
•

Write program logic or source code for each measure that identifies, tracks, and links
enrollment within and across product lines (Medicaid and CHIP), by age and sex, as well as

308 | PROTOCOL SEVEN

through possible periods of enrollment and disenrollment, which complies with the
specifications of each performance measure
•

Conduct tests of data to assess completeness, integration, and integrity, and to ensure there is
no double-counting of services reported through different data systems or suppliers

•

Assure that all enrollees who were eligible to receive the specified services were included in
the initial population from which the final denominator was produced. The eligible
population will include both enrollees who received the services and those who did not. This
same activity applies to provider groups or other relevant populations identified in the
specifications of each performance measure

Step 4: Conduct Preliminary Analysis
The EQRO will assess the completeness, accuracy, and reasonableness of the data in its
repository, and work with the MCP and other data suppliers until the data are satisfactory.
Referring to the ISCA conducted for the MCP and other data suppliers, the EQRO will have
identified areas of potential weakness. These should be considered in conducting analyses of
missing data, data quality, and supplier data issues.
Missing Data. The EQRO will analyze its repository for evidence of missing data. To determine
completeness, the EQRO should compare its data with data from the state, from prior years, and
from similar populations. Based on findings from the ISCA, the EQRO may pursue specific
concerns such as missing beneficiaries; missing providers, provider locations, or provider types;
and missing services or service types. Knowledge of the data suppliers’ contractual relationships
with providers from the ISCA as well as knowledge of the expected magnitude of reporting will
help identify specific areas to investigate for missing data. The EQRO should be aware of
instances when the MCP was unable to submit data, when submitted data failed edits, and when
data were not resubmitted.
Other Data Quality Issues. The EQRO should analyze the data it has received to identify data
quality problems such as inability to process or retain certain fields. Some MCPs may lack the
capacity to capture or maintain all the data elements that are required for submission, such as
secondary diagnoses or procedure codes or some coding specificity.
Supplier Data Issues. Using the edit checks from Activity 2, Step 2, the EQRO should identify
problems in how data suppliers compiled and submitted their data to the EQRO.
Significant issues may affect the feasibility of calculating valid and reliable performance
measures. Before proceeding to Step 5, the EQRO should report significant issues to the state
and determine whether the data are suitable for calculating selected measures.

PROTOCOL SEVEN | 309

Step 5: Calculate the Denominators, Numerators, and Rates
Following the specifications provided by the state, the EQRO calculates performance measures
from its data repository. To do this, the EQRO must purchase or write and test source code to
properly apply all specifications to identify the denominator population. The EQRO must apply
specified continuous enrollment and other eligibility criteria and implement exclusions from the
denominator.
Once the EQRO has identified all eligible beneficiaries in the denominator for a measure, it must
apply the specifications to identify cases that qualify for inclusion in the numerator. Where
sampling is required such as for medical record review, the EQRO must follow the specifications
for selecting an appropriate sample. The EQRO should follow the medical record review process
outlined in Activity 2, Step 1 regardless of when in the measure calculation process the medical
record review takes place.

Activity 3: Report Results
Step 1: Report Preliminary Performance Measure Results
Before sharing performance measure results with the state, the EQRO should share its
preliminary findings with the MCPs to obtain their feedback about the accuracy of the results.
The report should include, at a minimum, the following elements for each performance measure:
•

Data source(s)

•

Method (administrative, medical record review, hybrid)

•

Denominator

•

Sample size (if relevant)

•

Administrative numerator events (if relevant)

•

Medical record numerator events (if relevant)

•

Calculated rate

•

Deviations from the measure specifications (if relevant)

To enable the MCP to understand and interpret the results, the report may also analyze MCP
performance in relation to external benchmarks or prior-year performance.
The EQRO should invite the MCP to offer comments and documentation to support correction of
any factual errors or to clarify results. The EQRO should provide a reasonable period of time for
the MCP to provide its comments. The EQRO should then recalculate measures based on the
comments, if necessary, and revise its findings where appropriate.

310 | PROTOCOL SEVEN

Step 2: Analyze Performance Measure Results
Using the final calculations from Activity 3 Step 1, the EQRO should conduct all analyses
required by the state. The results should be presented in a format prescribed by the state, or if the
state has not prescribed a format, in a way that facilitates the state’s intended use of the
performance measures. Decisions about the format include the balance between text, tables, and
graphics, as well as the level of detail. In addition, the EQRO should determine whether the data
should be analyzed for each MCP individually, or whether individual MCP results should also be
compared to results for other MCPs or in relation to external benchmarks.

Step 3: Submit a Final Report to the State
The EQRO will submit a final report containing the performance measure results, analyses, and
recommendations in the format prescribed by the state and in the time frame required. The
content of the final report should include the following elements:
•

A summary of the EQRO’s performance measurement activities, including documentation of
the activities performed

•

Work papers and detailed results of key steps of the measure calculation process, MCPspecific performance measure rates, and accompanying analyses

•

Discussion of areas of MCP strength and opportunities for improvement in both data
management and performance

•

Recommendations for improving MCP performance

The EQRO should submit a draft report on performance measure results to the state (separate
from the EQR technical report) and revise the final report based on feedback from the state.
END OF PROTOCOL 7

PROTOCOL SEVEN | 311

Worksheets for Protocol 7: Performance Measure
Calculation Tools
Instructions. Use these or similar worksheets to identify the performance measures to be calculated, document the
technical specifications for each measure, develop a master list of data elements, indicate the data source and any
known data quality issues, and specify the file format for the transmission of the required data elements. This tool
includes the following worksheets crosswalked to the applicable Activity and Step:
Worksheet Name

Protocol Activity and Step

Worksheet 7.1. List of Performance
Measures to be Calculated

Activity 1. Step 1. Define the Scope of the Validation

Worksheet 7.2. Companion Performance
Measurement Tool

Activity 1. Step. 1. Define the Scope of the Validation
Activity 1. Step 3. Conduct Detailed Review of Measures
Activity 2. Step 4. Complete the Detailed Review of Measures

Worksheet 7.3. Data Element Master
Checklist

Activity 1. Step 4. Initiate Review of Medical Record Data Collection

Worksheet 7.4. Data Availability and Data
Quality

Activity 2. Step 5. Prepare for the MCP Site Visit

Worksheet 7.5. Illustrative File Format for
Transmission of Claims Data

Activity 2. Step 1. Review Information Systems Underlying
Performance Measurement

Activity 3. Post-Site Visit Activities

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM □ LTSS
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

312 | PROTOCOL SEVEN

Worksheet 7.1. List of Performance Measures to be Calculated
This worksheet is used to identify the measures to be calculated, including the measure source, how frequently the
measure is reported, and the reporting deadline. Complete the worksheet for each measure to be calculated, and
adapt as needed. Please note if you use the HEDIS® or Child/Adult Core Set specifications. Also note if the measure
is homegrown.
NQF # (if
applicable)

Measure
Steward

Performance measure

Measure source

Reporting
frequency

Reporting
deadline

The list below is illustrative of the information to be included in the worksheet. Please include National Quality Forum
(NQF) number (if applicable), and measure steward.
NQF # (if
applicable)

Measure
Steward

Performance measure

Measure source

Reporting
frequency

Reporting
deadline

0038

NCQA

Childhood Immunization Status (CISCH)

HEDIS® MY 2020/
Child Core Set

Annual

June 30th

1407

NCQA

Immunizations for Adolescents (IMACH)

HEDIS® MY 2020/
Child Core Set

Annual

June 30th

1392

NCQA

Well-Child Visits in the First 30
Months of Life (W30-CH)

HEDIS® MY 2020/
Child Core Set

Annual

June 30th

PROTOCOL SEVEN | 313

Worksheet 7.2. Companion Performance Measurement Tool
This worksheet is used to document the technical specifications and benchmarks for each measure to be calculated.
Complete the worksheet for each measure listed in Worksheet 7.1, and adapt as needed.
Measure Name and Description __________________________________________________________________
Characteristics

Measure specifications

Measure purpose (check all that
apply)







QI or PIP
Demonstration/waiver program (e.g., 1115 demonstration)
Pay for performance/Value-based purchasing
Public reporting
Other (specify) ____________________

Data collection method (check
one)







Administrative
Medical Record Review
Hybrid (administrative supplemented by medical record review)
Survey
Other (specify) ____________________

Sampling method (if applicable)

Specifications for sample size, sampling method and replacement methods:
____________________________________________________________

Age

Lower age limit: ____________________
Upper age limit: ____________________

Sex (check one)

 Males only
 Females only
 Males and females

Continuous enrollment

 No
 Yes (specify) ________________________

Index event (e.g., birthday;
discharge; prescription;
diagnosis; procedure)

 No
 Yes (specify) ________________________

Denominator elements and data
sources (e.g., member ID, age,
gender, enrollment and
disenrollment dates, diagnoses,
procedures)

A list of each data element needed to establish eligibility for the denominator:
_____________________________________________

Numerator elements and data
sources (e.g., procedure codes,
diagnosis codes, pharmacy
codes, lab results, dates of
service)

A list of each data element needed to establish eligibility for the numerator:
_____________________________________________

Denominator

Denominator statement:
_______________________________________________________________

For each denominator element, the allowable data source(s):
_____________________________________________

For each numerator element, the allowable data source(s):
_____________________________________________

Inclusions/exclusions:
_______________________________________________________________
Denominator time window:
_______________________________________________________________

314 | PROTOCOL SEVEN

Characteristics

Measure specifications

Numerator

Numerator statement:
_______________________________________________________________
Inclusions/exclusions:
_______________________________________________________________
Numerator time window:
_______________________________________________________________

Rate calculation

Formula for calculation of rate:
_________________________________________________________

Benchmark(s) (check all that
apply)

 State-level (specify):
____________________________________________________________
 Regional (specify):
____________________________________________________________
 National (specify):
____________________________________________________________
 Other (specify):
____________________________________________________________
Source(s):
____________________________________________________________

Other analysis requirements
(e.g., change from prior year or
comparison to state average or
best in state, including
statistical tests)

List required analyses:
_____________________________________________________________

PROTOCOL SEVEN | 315

Worksheet 7.3. Data Element Master Checklist
This checklist is used to develop a master list of data elements needed to calculate each performance measure.
Indicate whether each data element is required to calculate the measure, and adapt as needed.
Performance
Measure 1
Denominator
data elements:
Date of birth
Sex
Enrollment date
Disenrollment
date
Diagnosis code
Procedure code
Service date
Provider ID
Numerator data
elements:
Diagnosis code
Procedure code
Pharmacy code
Lab order
Lab result

316 | PROTOCOL SEVEN

Performance
Measure 2

Performance
Measure 3

Performance
Measure 4

Performance
Measure 5

Worksheet 7.4. Data Availability and Data Quality
This worksheet is used to document the data availability and data quality issues for each data element required to
calculate the denominator and numerator for each measure in Worksheet 7.1.

Available data
source(s)?

In data
repository?

Identified data quality issues (e.g.,
data completeness, integration
issues)

Denominator data elements:
Date of birth

 Yes
 No

Sex

 Yes
 No

Enrollment date

 Yes
 No

Disenrollment date

 Yes
 No

Diagnosis code

 Yes
 No

Procedure code

 Yes
 No

Service date

 Yes
 No

Provider ID

 Yes
 No

[INSERT other denominator
data elements]

 Yes
 No

Numerator data elements:
Diagnosis code

 Yes
 No

Procedure code

 Yes
 No

Pharmacy code

 Yes
 No

Lab order

 Yes
 No

Lab result

 Yes
 No

[INSERT other numerator data
elements]

 Yes
 No

PROTOCOL SEVEN | 317

Worksheet 7.5. Illustrative File Format for Transmission of Claims Data
This worksheet provides a template for constructing an electronic data shell or file format, including definitions for all
data fields. Complete this template for each data field in the claims file, and adapt as needed.

Field #

Applies to
UB Phys Rx

Data Field

Type/Format

Req/Opt

Comments

Row Type

Char(1)

Required

1=UB, 2=Phys, 3=Rx

Claim Status

Char(1)

Required

P=Paid, D=Denied
Denied claims are
highly desirable for
accurate performance
measurement

Recipient ID

Varchar(50)

Required

Medicaid or CHIP
identifier supplied by
the State for the
member. Native or
encrypted. If encrypted,
separate encryption key
must be provided.

Claim Number

Varchar(80)

Required

Required if source is
not sending final-only
versions of claims

Prior Version
Claim Number

Varchar(80)

Required

Required if source is
not sending final-only
versions of claims

Claim Received
Date

yyyymmdd

Required

Required if source is
not sending final-only
versions of claims

Claim Paid Date

yyyymmdd

Required

Required if source is
not sending final-only
versions of claims

Billing Provider ID

Varchar(30)

Required

Any internal identifier
for the billing provider.
Must be unique to one
clinician or entity. Must
exist on the provider
file. If supplying for Rx,
use pharmacy provider
ID.

Principal
Diagnosis

Varchar(5)

Required

No periods, left justified

Diagnosis 2

Varchar(5)

Required

No periods, left justified

Diagnosis 3

Varchar(5)

Required

No periods, left justified

END OF WORKSHEETS FOR PROTOCOL 7

318 | PROTOCOL SEVEN

Protocol 8. Implementation of Additional
Performance Improvement Projects
An Optional EQR-Related Activity

ACTIVITY 2: DEFINE THE PIP AIM STATEMENT
ACTIVITY 3: IDENTIFY THE PIP POPULATION
ACTIVITY 4: USE SOUND SAMPLING METHODS
ACTIVITY 5: SELECT THE PIP VARIABLES
ACTIVITY 6: COLLECT VALID AND RELIABLE DATA
ACTIVITY 7: ANALYZE DATA AND INTERPRET RESULTS
ACTIVITY 8: REVIEW IMPROVEMENT STRATEGIES
ACTIVITY 9: ASSESS WHETHER SIGNIFICANT AND SUSTAINED
IMPROVEMENT OCCURRED

Background
Federal regulations at 42 CFR 438.330(b)(1) and 457.1240(b) require that
Medicaid and CHIP managed care plans (MCPs) conduct performance
improvement projects (PIPs) that focus on both clinical and nonclinical areas as
part of a comprehensive quality assessment and performance improvement
(QAPI) program. 77 Validation of the PIPs conducted by MCPs as a part of their
QAPI programs is a mandatory external quality review (EQR)-related activity,
as described in Protocol 1. Validation of Performance Improvement Projects.
In addition, Federal regulations at 42 CFR 438.358(c)(4) and 457.1250(a)
specify that the external quality review organization (EQRO) may conduct PIPs
in addition to those performed by the MCPs as a part of their QAPI programs.
These additional PIPs are an optional EQR-related activity. These PIPs can be
conducted in conjunction with CMS and state quality improvement priorities or
to align with national quality improvement initiatives.

PROTOCOL EIGHT

ACTIVITY 1: SELECT THE PIP TOPIC

At a minimum, a single PIP that focuses on both clinical and non-clinical aspects of care may satisfy
this requirement. Otherwise, a state must require at least two PIPs, one clinical and one non-clinical.

319

These activities are eligible for enhanced Federal financial participation at a 75 percent rate (1)
when conducted by a qualified EQRO on an MCO, (2) when the EQR-related activities are
completed using methodologies consistent with the protocols contained within this document,
and (3) when the state receives approval of its EQRO contract from CMS. 78, 79
This protocol provides guidance to states on the implementation of additional (non-QAPI) PIPs
using EQROs to assess and improve processes and outcomes of care provided by MCPs in the
state.

Getting Started on Protocol 8
Protocol 8 consists of nine activities for implementing additional PIPs (Figure 8.1).

If the state or the state’s agent that is not an MCP conducts the EQR-related activity on an MCO, it would be eligible for the
50 percent match rate. See 42 CFR 438.370(a)–(b).

320 | PROTOCOL EIGHT

Figure 8.1. Protocol 8 Activities

PROTOCOL EIGHT | 321

As shown in Table 8.1, these activities align with the nine steps in Activity 1, Protocol 1,
Validation of Performance Improvement Projects. To streamline the content in this protocol, and
avoid duplication with Protocol 1, please refer to the relevant sections in Protocol 1 and the
associated worksheets.
Table 8.1. Crosswalk between Protocol 8, PIP Implementation and Protocol 1, PIP
Validation
Protocol 8. PIP Implementation
Activities

Protocol 1. PIP Validation
Activity 1 Steps

Protocol 1. PIP Validation
Tool Worksheets

Activity 1. Select the PIP Topic

Step 1. Review the Selected PIP Topic

Worksheet 1.1

Activity 2. Define the PIP Aim
Statement

Step 2. Review the PIP Aim Statement

Worksheet 1.2

Activity 3. Identify the PIP Population

Step 3. Review the Identified PIP Population

Worksheet 1.3

Activity 4. Use Sound Sampling
Methods

Step 4. Review the Sampling Method

Worksheet 1.4

Activity 5. Select the PIP Variables

Step 5. Review the Selected PIP Variables
and Performance Measures

Worksheet 1.5

Activity 6. Collect Valid and Reliable
Data

Step 6. Review the Data Collection
Procedures

Worksheet 1.6

Activity 7. Analyze Data and Interpret
Results

Step 7. Review Data Analysis and
Interpretation of PIP Results

Worksheet 1.7

Activity 8. Review Improvement
Strategies

Step 8. Assess the Improvement Strategies

Worksheet 1.8

Activity 9. Assess Whether Significant Step 9. Assess the Likelihood that Significant
and Sustained Improvement Occurred and Sustained Improvement Occurred

Worksheet 1.9

Three supplemental resources are available to help EQROs design and implement additional
PIPs which can lead to significant and sustained improvement in health care delivery processes
and outcomes:
•

Protocol 1. Validation of Performance Improvement Projects

•

Worksheets for Protocol 1. PIP Validation Tools and Reporting Framework

•

Appendix B. Sampling Approaches for EQR Data Collection Activities

Activity 1: Select the PIP Topic
Additional PIP topics should target improvement in clinical and/or nonclinical services provided
to Medicaid and CHIP enrollees in the state. Selected topics should reflect the characteristics of
Medicaid and CHIP enrollees in terms of demographics, prevalence of disease, and the potential
consequences of the disease. It is recommended that PIP topics align with: (1) the National

322 | PROTOCOL EIGHT

Quality Strategy, 80 (2) the CMS Quality Strategy, 81 and (3) the state’s managed care quality
strategy. The EQRO should also review the state’s performance on the CMS Child and Adult
Core Set measures to identify measures whose performance could be impacted by a managed
care PIP. 82 Additional PIPs can be used to help drive improvement on these measures. For more
information about selecting a PIP topic, see Protocol 1, Activity 1, Step 1.

Activity 2: Define the PIP Aim Statement
The PIP aim statement identifies the focus of the PIP and establishes the framework for data
collection and analysis. The PIP aim statement should define the improvement strategy,
population, and time period. It should be clear, concise, and answerable. When identifying the
PIP aim statement, potential sources of information include:
•

State performance on the Child and Adult Core Sets

•

State data relevant to the topic being studied

•

MCP data relevant to the topic being studied

•

Enrollee focus groups or surveys

•

Relevant clinical literature on recommended care and external benchmarks

For more information about developing the PIP aim statement, see Protocol 1, Activity 1, Step 2.

Activity 3: Identify the PIP Population
The additional PIP must clearly identify the target population in relation to the PIP aim statement
(such as age, length of enrollment, diagnoses, procedures, and other characteristics). Depending
on the nature of the PIP aim statement, population, and available data, the PIP may include the
entire population or a sample of the population. PIPs that rely on existing administrative data,
such as claims and encounter data, registry data, or vital records are typically based on the
universe of the PIP’s population. PIPs that require medical record review typically include a
representative sample of the identified population. If a sample is used, go to Activity 4. If the
entire population will be studied, skip Activity 4 and go to Activity 5. If HEDIS® measures and
sampling methodology are used, go to Activity 5. For more information about identifying the PIP
population, see Protocol 1, Activity 1, Step 3.

More information about the HHS National Quality Strategy is available at
https://www.ahrq.gov/workingforquality/about/index.html.

More information about the CMS Quality Strategy is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityInitiativesGenInfo/CMS-Quality-Strategy.html.

PROTOCOL EIGHT | 323

Activity 4: Use Sound Sampling Methods
Appropriate sampling methods are necessary to ensure the collection of information that
produces valid and reliable results. Refer to Appendix B, Sampling Approaches for EQR Data
Collection Activities, for an overview of sampling methodologies applicable to PIPs. When
HEDIS® measures are used and sampling is required (e.g., for measures calculated using the
hybrid method), HEDIS® sampling methodology should be used. For more information about
sampling approaches for EQR activities, see Protocol 1, Activity 1, Step 4, and Appendix B.

Activity 5: Select the PIP Variables
The next step is to select the PIP variables. Variables can take a variety of forms as long as the
selected variables measure performance on the PIP aim statement objectively and reliably and
use clearly defined indicators of performance. The additional PIP should include the number and
type of PIP variables that are adequate to answer the PIP aim statement and for which
appropriate and reliable data are available to measure performance and track improvement over
time. Data availability should also be considered when selecting PIP variables, as more frequent
access to data, such as on a monthly, quarterly, or semi-annual basis, supports continuous quality
improvement (QI) and Plan Do Study Act (PDSA) efforts and can allow and MCP or state to
correct or revise course more quickly, if needed. CMS encourages states to select PIP variables
and performance measures that can be examined on at least a semi-annual basis.
To the extent possible, CMS encourages EQROs to choose variables for PIPs that reflect health
outcomes. Performance measures are then used to measure these outcomes. When selecting
measures for an additional PIP, first consider existing performance measures because the
specifications for these measures often have been refined over time, may reflect current clinical
guidance, and may have benchmarks for assessing performance. CMS encourages the use of the
following existing performance measure sets: Child and Adult Core Set measures, behavioral
health clinic quality measures, and Core Quality Measures Collaborative. 83 Other examples of
existing measures include NCQA’s Healthcare Effectiveness Data Information Set (HEDIS®) or
measures that have been developed by AHRQ (such as the prevention quality indicators,
inpatient quality indicators, patient safety indicators, and pediatric quality indicators). 84

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about measures for behavioral health clinics is available at https://www.samhsa.gov/section-223/quality-measures. More
information about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-InitiativesPatient-Assessment-Instruments/QualityMeasures/Core-Measures.html.

More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.

324 | PROTOCOL EIGHT

When there are gaps in existing measures, new measures may need to be developed based on
current clinical practice guidelines or health services research. Consider the following questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study question?

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the measure to be calculated reliably and accurately? Are
there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

For more information about the selection of PIP variables, see Protocol 1, Activity 1, Step 5.

Activity 6: Collect Valid and Reliable Data
Data collection procedures during implementation of the PIP must ensure that the data used to
measure performance are valid and reliable. Validity means that the data are measuring what is
intended to be measured. Reliability means that the data are producing consistent results.
To ensure validity and reliability of the data collected as part of implementing the PIP, the data
collection plan should specify:
•

The data sources for the PIP

•

The data to be collected

•

How and when the data are to be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments used to collect the data

The PIP may involve two main kinds of data collection: administrative data sources and medical
record review. Procedures to collect data from administrative data systems will be different from
procedures for visual inspection of medical records or other primary source documents.
However, both types of data collection require assurances that data are valid and reliable. For
more information about assuring the validity and reliability of PIP data collection procedures, see
Protocol 1, Activity 1, Step 6.

Activity 7: Analyze Data and Interpret Results
Data analysis begins with assessing performance on the selected clinical or nonclinical measures
using appropriate statistical techniques, as specified in the data analysis plan. Interpretation and
analysis of the PIP data should be based on a continuous improvement philosophy and reflect an
PROTOCOL EIGHT | 325

understanding of lessons learned and opportunities for improvement. Interpretation of the PIP
results should involve assessing the causes of less-than-optimal performance and collecting data
to support the assessment. Accurate data analysis, including measurements at multiple points in
time and tests for statistical significance, is essential because the state or MCP may implement
changes based on the results. For more information on data analysis and interpretation of PIP
results, see Protocol 1, Activity 1, Step 7.

Activity 8: Review Improvement Strategies
Building on the data analysis and interpretation of PIP results in Activity 7, the next step is to
review the improvement strategies implemented as part of the PIP. Significant, sustained
improvement is the result of developing and implementing effective improvement strategies
(including strategies that are culturally and linguistically appropriate for the target population).
Selected strategies should be evidence-based, that is, there should be existing evidence
(published or unpublished) suggesting that the test of change would be likely to lead to the
desired improvement in processes or outcomes (as measured by the PIP variables). The
effectiveness of the improvement strategy is determined by measuring change in performance
according to predefined measures.
CMS encourages states to work with their EQROs and MCPs to determine which PIP
methodology best suits the needs of the state, its MCPs, and their beneficiaries. For example, a
state may use the Institute for Healthcare Improvement’s (IHI) Model for Improvement to guide
improvement work and test changes on a small scale using Plan Do Study Act (PDSA) cycles
(see Protocol 1 for more information on the Model for Improvement and PDSA cycles) 85 PDSA
cycles provide a methodology to test changes on a small scale and to apply rapid-cycle learning
principles to adjust intervention strategies over the course of the improvement. This approach
involves a continuous cycle of measuring and analyzing performance and requires frequent
reflection and course correction. Data should be evaluated on a regular basis and interventions
should be adjusted based on what was learned. Interventions can then be scaled to larger settings
or populations if found effective. PIPs, based on the Model for Improvement and PDSA process
are sometimes known as rapid-cycle PIPs. For more information on the use of the IHI Model for
Improvement and PDSA cycles, see Protocol 1, Activity 1, Step 8.

Activity 9: Assess Whether Significant and Sustained
Improvement Occurred
A PIP is intended to result in significant and sustained improvement in health care delivery
processes and outcomes, rather than short-term or random change. The final activity in the PIP is

More information about the Model for Improvement and PDSA approach is available from the following sources:
http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx and
https://innovations.ahrq.gov/qualitytools/plan-do-study-act-pdsa-cycle.

326 | PROTOCOL EIGHT

to assess whether the PIP resulted in statistically significant changes over time that could
reasonably be attributed to the improvement strategy implemented as part of the PIP.
To assess whether significant and sustained improvement occurred, repeated measurements are
required, using the same methodology used for the baseline measurement. In addition, tests of
statistical significance are required to assess whether there is evidence of statistically significant
improvement. For more information on assessing the likelihood that significant and sustained
improvement occurred, see Protocol 1, Activity 1, Step 9.
END OF PROTOCOL 8

PROTOCOL EIGHT | 327

Protocol 9. Conducting Focus Studies of
Health Care Quality
An Optional EQR-Related Activity

ACTIVITY 2: DEFINE THE STUDY QUESTION(S)
ACTIVITY 3: SELECT THE STUDY VARIABLE(S)
ACTIVITY 4: DEVELOP A PLAN TO STUDY THE POPULATION
ACTIVITY 5: COLLECT DATA
ACTIVITY 6: ANALYZE AND INTERPRET STUDY RESULTS
ACTIVITY 7: REPORT RESULTS TO THE STATE

Background
States may direct their external quality review organizations (EQROs) to
conduct focus studies for quality improvement (QI), administrative, legislative,
or other purposes. Similar to performance improvement projects (PIPs), focus
studies may examine clinical or nonclinical aspects of care provided by
managed care plans (MCPs). However, there are key differences between focus
studies and PIPs (see Box 9.1). Focus studies assess quality of care at a point in
time, whereas PIPs assess improvement over time. For example, a focus study
may be conducted for a single year to provide the state with information about
the baseline status of health care quality for a particular aspect of care across
managed care in the state or for subpopulations served by managed care within
the state. By comparison, PIPs are designed to achieve significant improvement,
sustained over time, in health outcomes and enrollee experience. PIPs include
the implementation of interventions to achieve improvement, evaluation of the
intervention’s effectiveness (including performance measurement), and
initiation of activities to increase or sustain improvement.

PROTOCOL NINE

ACTIVITY 1: SELECT THE STUDY TOPIC(S)

Box 9.1. How Does a Focus Study Differ from a PIP?
A Focus Study is a study of a particular aspect of clinical care or nonclinical services
provided by an MCP at a point in time.
A PIP is a project that implements an intervention designed to achieve and sustain
significant improvement over time.

328

Although the goals and regulations for focus studies and PIPs differ, EQROs can use similar
processes to design both types of projects. Both must be designed, conducted, and reported in a
methodologically sound manner. Because of these similarities, the process for conducting focus
studies described in this protocol mirrors many of the activities in Protocol 8 for conducting
PIPs.

Getting Started on Protocol 9
To complete this protocol, the EQRO undertakes seven activities for each MCP (Figure 9.1).
Figure 9.1. Overview of Protocol 9 Activities

PROTOCOL NINE | 329

Activity 1: Select the Study Topic(s)
Focus studies may target a single MCP, a subset of MCPs, or all MCPs in the state. They should
target relevant areas of clinical care and nonclinical services in which it is known or suspected
that improvement is needed. For example, the focus study may examine patterns of over- or
under-utilization of services to assess the potential threat to health or functional status of
enrollees. Selected topics should:
•

Reflect MCP enrollment in terms of demographic characteristics, prevalence of disease, and
the potential consequences of the disease

•

Affect a significant portion of enrollees, a specified subpopulation of enrollees, or a
significant portion of enrollees impacted by a specific health care issue (such as oral health or
maternal and infant health)

•

Align with priority areas as identified in the HHS and/or CMS quality strategies. 86

When selecting the focus study topic, the EQRO and state should consider a variety of factors
related to enrollee characteristics, health risks, experience of care, and special population or
service needs. Box 9.2 provides additional information on selecting a focus study topic.
Box 9.2 Factors to Consider when Selecting a Focus Study Topic
• Demographic and epidemiologic information about current MCP enrollees
• Enrollee health risks and disease prevalence
• State performance on CMS Child and Adult Core Set measures
• Input from enrollees about specific services, such as mental health or substance abuse
• A spectrum of enrollee populations, services, and experiences:
○ Care for children with special health care needs
○ Care for adults with physical disabilities
○ Care for people with intellectual and developmental disabilities
○ Care for people with dual eligibility who use long-term services and supports (LTSS)
○ Preventive care
○ Acute and chronic care
○ High-volume and high-risk services (even if they are low frequency)
○ Specialized care received from centers (burn, transplant, and cardiac surgery centers)
○ Access to and availability of care
○ Continuity or coordination of care from multiple providers and over multiple episodes
○ Appeals and grievances

More information about the HHS Quality Strategy is available at https://www.ahrq.gov/workingforquality/about/index.html.
More information about the CMS Quality Strategy is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityInitiativesGenInfo/CMS-Quality-Strategy.html.

330 | PROTOCOL NINE

Activity 2: Define the Study Question(s)
In this activity, the EQRO defines the study question(s). The study question identifies the focus
of the study and establishes the framework for data collection and analysis. The study question
should be clear, concise, and answerable. Box 9.3 provides considerations for the study
question(s). Table 9.1 critiques illustrative study questions for a focus study.
Box 9.3 How do we know if a Focus Study Question is Clear, Concise, and Answerable?
A good focus study question specifies measurable indicators and analytics for a defined population and time period.
Potential sources of information to help form the study question include:
• State data relevant to the topic being studied
• MCP data relevant to the topic being studied
• CMS Child and Adult Core Set measures
• Enrollee focus groups or surveys
• Relevant clinical literature on recommended care and external benchmark

Table 9.1. Examples of Focus Study Questions
Illustrative Study Questions
Poor Study What is the status of preventive dental care in
Question
Medicaid?

Critique
• Does not specify measurable indicators and
analytics
• Does not define the population and time period

Good
Study
Question

How does the rate of preventive dental visits
among children enrolled in Medicaid for at least
six months in calendar year 2021 vary by age,
geographic location, and race/ethnicity?

• Specifies the measurable indicator (preventive
dental visits)
• Specifies the analytic issue of interest (variation
in utilization rates by age, geographic location,
and race/ethnicity)
• Defines the population and time period (children
enrolled in Medicaid for at least six months in
calendar year 2016)

Activity 3: Select the Study Variable(s)
In this activity, the EQRO selects the study variable(s) (see Box 9.4). Study variables can take a
variety of forms as long as the selected variables identify the MCP’s performance on the study
questions objectively and reliably and use clearly defined measurable indicators of performance.
The study variables for the focus study should allow the EQRO to measure the MCP’s
performance on the elements of care identified in the study question(s). For example, for a focus
study on preventive dental services in an MCP, the EQRO should select one or more study
variables that directly assess enrollees’ access to and use of these services. Examples of such
study variables may include:

PROTOCOL NINE | 331

•

The proportion of eligible enrollees who received any preventive dental service within a
defined timeframe

•

The proportion of eligible enrollees who received a dental sealant within a defined timeframe

•

The ratio of dental service providers providing preventive services per 1,000 MCP enrollees
within a defined geographic area

•

The proportion of MCP enrollees who report being able to obtain preventive dental services
within a specified timeframe in an enrollee survey
Box 9.4 What is a Study Variable?
A study variable is a measurable characteristic, quality, trait, or attribute of a particular individual, object, or situation
being studied. When selecting study variables, consider different types of variables and choose the variables that are
best suited to the available data, resources, and study questions.

The EQRO should choose the number and type of study variables that are adequate to answer the
study question(s) and for which appropriate and reliable data are available to determine the state
of the population at a point in time. Study variables may be continuous, categorical, or discrete
(Table 9.2), and use a variety of measurement scales to assess performance (Table 9.3).
Table 9.2. Types of Variables for Focus Studies
Variable Type
Continuous

Example
• Age, blood pressure, temperature,
height/weight, body mass index,
birthweight

Categorical

Have a range of non-ordered, qualitative values (or
categories)

• An enrollee survey question that asks
enrollees to identify the most important
among a list of incentives offered to
improve well-care visit rates

Discrete

Have a limited number of possible categories

• An enrollee has/has not received a flu
shot in the past 12 months

Note: binary variables have two categories

Table 9.3. Types of Measurement Scales for Focus Studies
Measurement
Scales
Interval

Definition
The distances between numbers denote significant
and interpretable differences (e.g., dollars, degrees,
inches, pounds) and the differences are interpretable
as higher or lower.

332 | PROTOCOL NINE

Example
• The interval between an annual income
of $40,000 and $30,000 = $10,000

Measurement
Scales

Definition

Example

Ordinal

Can be treated as quantitative in some
circumstances, and qualitative in others

• An enrollee survey question that asks
enrollees to rank their experience of
care on a scale from 1 (low quality) to 5
(high quality)

Nominal

The set of categories for a qualitative variable

• Mode of transportation to work (car,
bus, subway, bicycle, walk)

When selecting study variables, the EQRO should consider measures that currently exist within
the health services research community or the managed care industry because the specifications
for these measures often have been refined over time, may reflect current clinical guidance, and
may have benchmarks for assessing MCP performance. CMS encourages use of the CMS Child
and Adult Core Set measures, behavioral health clinic quality measures, and Core Quality
Measures Collaborative for examples. 87 Additional examples of existing measures include
NCQA’s Healthcare Effectiveness Data Information Set (HEDIS®) or measures that have been
developed by AHRQ (such as the prevention quality indicators, inpatient quality indicators,
patient safety indicators, and pediatric quality indicators). 88
When there are gaps in existing measures, the EQRO may develop new measures based on
current clinical practice guidelines or health services research. The EQRO should consider the
following questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study question?

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the EQRO to reliably and accurately calculate the
measure? Are there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about measures for behavioral health clinics is available at https://www.samhsa.gov/section-223/quality-measures. More
information about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-InitiativesPatient-Assessment-Instruments/QualityMeasures/Core-Measures.html.

More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.

PROTOCOL NINE | 333

Activity 4: Develop a Plan to Study the Population
The focus study should be designed to assess performance for all eligible enrollees in the
population being studied (e.g., a single MCP, all MCPs in the state, or all Medicaid beneficiaries
in the state). If the study focuses on a specific service (e.g., preventive dental care) or condition
(e.g., diabetes), the EQRO should include all members of the population that meet measurespecific eligibility criteria (see Box 9.5). For example, in a focus study on human papillomavirus
(HPV) vaccine rates, the EQRO should specify the population of MCP enrollees who meet the
eligibility criteria based on age or other factors.
Box 9.5 Tips for Defining the Study Population
EQROs should carefully review the technical specifications for existing measures or develop detailed specifications for
new measures to define the measure-eligible population included in the denominator for the study variables.

Once the study population is defined, the EQRO should decide whether to review performance
for every enrollee in the study population or whether performance needs to be assessed for a
representative sample of the population (see Box 9.6). The appropriate method for the focus
study may depend on factors related to the availability and quality of data.
Box 9.6. Factors to Consider in Deciding Whether to Study the Total Population or a Sample
• If the information needed for a study variable is reliable and complete for the population in an available data source
(such as claims/encounters or enrollment data), measuring the total population might be appropriate
○ For example, if the MCP's encounter data reliably provide information about the use of prenatal care (e.g., date,
type of visit, type of provider), then the EQRO can use these records to assess performance for the total study
population
• If the EQRO has concerns about the reliability and completeness of administrative data, it may be more appropriate
to conduct the study for a sample of the population
○ For example, a study on childhood obesity that requires data on body mass index (BMI) assessments may not be
able to obtain this information in existing administrative data and may need to use medical records for a sample
of enrollees
○ Similarly, a study on prenatal and postpartum care may need to use medical records where an MCP pays for
maternity care using a bundled payment approach; in such cases, administrative data may not provide the
necessary level of detail on the number and date of prenatal and postpartum care visits

If the EQRO decides to assess performance for a sample of the study population, it should use
standard statistical methods to select the sample. The sample should be drawn in a way that
ensures that it will reflect the total study population. If the sample is not representative, then the
focus study will not produce valid and reliable (generalizable) results. To ensure that the sample
is representative, every enrollee in the study population should have an equal probability of
being selected for the sample and the sample should be large enough to reflect the diversity in
the study population. If the focus study results will be assessed for particular subpopulations of
enrollees, the sampling strategy should be designed so that there are sufficient numbers of
334 | PROTOCOL NINE

enrollees in each subpopulation included in the sample. Please refer to Appendix B for additional
guidance on sampling approaches and sample sizes.

Activity 5: Collect Data
Once the EQRO has established the focus study topic, question(s), variable(s), and population,
the next step is to collect data. Because data collection can be costly and burdensome, the EQRO
should first develop a data collection plan that specifies:
•

The data sources for the focus study

•

The data to be collected

•

How and when the data will be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments that will be used to collect the data

The plan should clearly specify the data sources and explain how the EQRO will ensure that the
collected data are complete and reliable for the total study population. For example, if the EQRO
will analyze claims or encounter records for an MCP, the EQRO should assess the data to ensure
that it contains consistent and complete information for all enrollees included in the study
population. Moreover, the EQRO should ensure that diagnosis and procedure codes are used
consistently across providers and that services provided in all settings are included.
If the EQRO plans to develop original data collection tools for the focus study, these tools should
be designed to obtain reliable results for all subpopulations included in the study. In addition, the
data collection plan should confirm that the individuals conducting the data collection have the
necessary expertise and training for the task. For example, if the focus study requires a survey,
interviewers should be trained to conduct the survey in a systematic, unbiased manner. Similarly,
if the focus study requires medical record reviews, special attention should be given to the
qualifications of the medical record reviewers, the specificity of the guidelines for data
collection, and plans for ensuring inter- and intra-rater reliability. The reviewers should have a
standard protocol for reviewing records, have the knowledge to interpret the records, and have
been trained to identify and code the information in the records using consistent decision rules.
Box 9.7 identifies considerations for medical record reviews.

PROTOCOL NINE | 335

Box 9.7. Considerations for Medical Record Reviews
• Medical record reviewers require the conceptual and organizational skills to abstract data. These skills will vary
depending on the nature of the data and the degree of professional judgment required. For example, experienced
clinical staff (such as registered nurses) should be used to extract the appropriate data from medical records to
support a judgment about whether clinical criteria are met. In contrast, trained medical assistants or medical records
clerks may collect data if the abstraction involves verifying the presence of a diagnostic test report
• Guidelines for obtaining and recording the data are essential. A glossary of terms should be developed before data
collection begins to ensure consistent interpretation among and between reviewers. In addition, reviewers should
have clear and succinct written instructions, including an overview of the study, how to complete each section of the
form, and general guidance on how to handle situations not covered by the instructions. This is particularly important
when multiple reviewers are collecting data
• Plans for ensuring inter- and intra-rater reliability are key. The number of reviewers used for a given project affects
the reliability of the data. A smaller number of staff promotes inter-rater reliability; however, it may also increase the
amount of time it takes to complete the task. The focus study should also consider and address intra-rater reliability
(i.e., reproducibility of judgments by the same abstractor at a different time)

Activity 6: Analyze and Interpret Study Results
Before beginning the focus study, the EQRO should establish a plan for analyzing and
interpreting the data. Data analysis begins with examining performance on the selected clinical
or nonclinical indicators. Accurate data analysis is essential because the state and MCPs may
implement changes in treatment and operations based on the results. The review should be
conducted using statistical analysis techniques defined in the data analysis plan.
Interpretation and analysis of the study data should involve developing hypotheses about the
causes of less-than-optimal performance and collecting data to validate the hypotheses.
Interpretation and analysis should also be based on a continuous improvement philosophy to
identify areas for improving administrative or delivery system processes.
The EQRO should conduct a quality assurance (QA) review of the analysis before it is finalized.
When reviewing the data analysis and study results, the QA reviewer should consider the
following questions:
•

Was the analysis conducted in accordance with the data analysis plan? Were conventional
methods used to conduct the analysis?

•

Are numerical results and findings presented in an accurate, clear, and easily understood
manner?

•

Does the analysis identify:

The focus study question and variables used to address the question?

Performance by key subgroups (e.g., by age, geographic location, health status, MCP)?

Realistic and unambiguous targets/benchmarks for the measures?

Statistical significance of differences among subgroups?

336 | PROTOCOL NINE

o
•

Factors that threaten the validity and reliability of the findings (e.g., missing data)?

Does the analysis of the study data include an interpretation of the extent to which the focus
study is successful and what follow-up activities are planned as a result?

Activity 7: Report Results to the State
After completing the focus study (including the QA review), the EQRO will report the results to
the state in its EQR technical report (see Box 9.8). Because the state may use the report to meet
its reporting requirements to federal or state agencies, the state legislature, local advocacy
groups, or other interested parties, the state may need the report to include specific information
presented in a specific format. At minimum, the report should include the following information
about the focus study:
•

Overall summary of findings

•

Study question and objectives

•

Methods of data collection and analysis

•

Detailed findings, including tables and graphics

•

Conclusions drawn from the data

To ensure that the report includes appropriate information in the desired format, the EQRO
should submit an outline to the state before writing up the results. The EQRO should also
confirm the audience for the report and the plans for dissemination (e.g., to CMS, MCPs,
providers, advocates, state legislators). With this information, the report can be appropriately
targeted to the intended audience.
Box 9.8. Tips on Reporting on Focus Studies in EQR Technical Reports
• Define the study question and objectives, methods, and data sources clearly and completely. Specify the study
population and time frame
• Use tables and graphics to “tell a story” with the data; make sure to answer the study question with the data.
• If comparisons are made between subgroups, conduct tests of statistical significance to determine whether
differences are statistically meaningful
• Describe the implications of the findings for understanding current performance and for developing quality
improvement initiatives. Identify strengths and weaknesses related to quality, timeliness, and access.
• Clearly state the study limitations and caveats

END OF PROTOCOL 9

PROTOCOL NINE | 337

Protocol 10. Assist With Quality Rating of
Medicaid and CHIP Managed Care
Organizations, Prepaid Inpatient Health
Plans, and Prepaid Ambulatory Health Plans
An Optional EQR-Related Activity

PROTOCOL TEN

Reserved.

338

Appendix A. Information Systems
Capabilities Assessment
ACTIVITY 1: MCP COMPLETES THE ISCA TOOL
ACTIVITY 2: PERFORM PRELIMINARY ISCA REVIEW
ACTIVITY 3: CONDUCT MCP ONSITE OR VIRTUAL SITE VISIT

ACTIVITY 5: DRAFT ISCA SUMMARY FOR EQR TECHNICAL REPORT

Background
This appendix defines the recommended capabilities of a managed care plan’s
(MCP’s) 89 information system (IS) to meet regulatory requirements for
managed care quality assessment and reporting, and provides an approach the
external quality review organization (EQRO) can use to assess the strength of
each MCP’s information system capabilities. Portions of the Information
Systems Capabilities Assessment (ISCA) are voluntary; however, some
components are required for the mandatory EQR-related activities protocols.
The regulations at 42 CFR 438.242 and 457.1233(d) also require the state to
ensure that each MCP maintains a health information system that collects,
analyzes, integrates, and reports data for purposes including utilization, claims,
grievances and appeals, disenrollment for reasons other than loss of Medicaid or
CHIP eligibility, rate setting, risk adjustment, quality measurement, value-based
purchasing, program integrity, and policy development.
Figure A.1 shows the interrelationship of data activities for providers, MCPs,
and EQROs. Per 42 CFR 438.242, the MCP’s information system must be able
to achieve the following:

APPENDIX A

ACTIVITY 4: COMPILE AND ANALYZE ISCA FINDINGS

1. Provide the state with all data elements the state deems necessary for the
mechanized claims processing and information retrieval systems it uses for
the management, monitoring, and administration of its Medicaid or CHIP
program. Collect data on enrollee and provider characteristics as specified
by the state provider and eligibility files, and on all services received by an

For the purposes of the EQR protocols, the term MCP includes managed care organizations (MCOs),
prepaid inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs), but does not
include primary care case management (PCCM) entities.

enrollee regardless of payment methodology, including services sub-capitated by a MCP to a
provider, through an encounter data system or other method that meets state requirements
2. Ensure that data received from providers are accurate and complete by:

Verifying the accuracy and timeliness of reported data

Collecting data from providers in standardized formats (e.g., T-MSIS) to the extent
feasible and appropriate

Screening the data for completeness, logic, and consistency

3. Make all collected data available to the state and to CMS upon request
This appendix provides an overview of the activities for assessing an MCP’s data collection,
processing, and reporting systems. The appendix concludes with information about the future of
information system assessments.

A.2 | APPENDIX A

Figure A.1. Provider, MCP, and EQRO Data Activities

APPENDIX A | A.3

Getting Started on the ISCA
Assessing an MCP’s information systems encompasses five consecutive activities (Figure A.2).
In the event that onsite activities are not feasible due to the COVID-19 pandemic, site visits may
be conducted virtually.
Figure A.2. Overview of ISCA Activities

One supplemental resource is available to help EQROs conduct validation of the ISCA:
•

Worksheets for Appendix A. Information System Capabilities Assessment (ISCA) Tools,
which can be used to enable the MCP to collect standard information about its information
system, and to guide onsite or virtual information systems interviews of MCP staff

A.4 | APPENDIX A

Activity 1: MCP Completes the ISCA Tool
The MCP should complete the ISCA tool (Worksheet A.1) to
provide standard information about its IS and gather all
requested documentation identified on a checklist at the end
of the assessment tool. The MCP should return the completed
ISCA tool and documentation to the EQRO within a
timeframe defined by the state.
Some states assess the capabilities of the MCP’s information
system as part of pre-contracting, contract compliance, or
contract renewal activities. The MCP must make any
previously conducted assessments accessible to the EQRO.
The EQRO should review any such assessments as part of its
ISCA review process.

WORKSHEET A.1
Resource for Activities 1 & 2
Worksheet A.1. Information System
Capabilities Assessment (ISCA) Tool
• An information collection tool
provided to an MCP by the state or its
EQRO to obtain the information
needed to validate the capabilities of
the MCP’s information systems,
processes, and data, to support
annual EQR-related activities and
associated EQR analysis and
recommendations

Activity 2: Perform Preliminary ISCA
Review
The EQRO assesses the adequacy of MCP policies and procedures based on the information
submitted by the MCP on the ISCA tool (Worksheet A.1) and its accompanying documentation.
MCP answers should be evaluated against the information system standards established by the
state to calculate and report specific plan-level performance measures, and collect and submit
encounter data to the state. The EQRO should identify sections of the ISCA that the MCP has not
fully completed. The EQRO may use the Managed Care Plan (MCP) Information System
Review Worksheet & Interview Guide (Worksheet A.2) to organize information for the site visit
interviews with MCP staff (Activity 3).

APPENDIX A | A.5

Activity 3: Conduct MCP Onsite or Virtual Site Visit
The EQRO conducts an onsite or virtual site visit to the
MCP to validate the completed ISCA tool (Worksheet
A.1) and to gather additional information as needed. The
EQRO conducts interviews with MCP staff responsible
for completing the ISCA, as well as additional staff
responsible for the MCP’s information system functions.
The interviews focus on the topics outlined in the ISCA
Interview Guide (Worksheet A.2), based on the pre-site
analysis of the ISCA in Activity 2. The interview with the
MCP should be closely coordinated with the MCP site
visit performed in Protocol 3. Assessment of Compliance
with Medicaid and CHIP Managed Care Regulations.
Refer to Protocol 3, Activity 3 for steps in conducting a
successful MCP site visit.

Activity 4: Compile and Analyze
ISCA Findings

WORKSHEET A.2
Resource for Activity 3
Worksheet A.2. Information System
Review Worksheet & Interview Guide
• A tool to conduct interviews with MCP
staff that completed the ISCA tool
(Worksheet A.1), as well as other
MAP staff as needed
• These questions are intended to
guide the reviewer's discussion with
MCP staff to help validate the
completed ISCA
• The questions are first organized by
MCP staff roles and then by
regulatory provision

At the conclusion of the ISCA site visit, the EQRO compiles and analyzes the information
gathered through the preliminary ISCA review (Activity 2) and from the site visit (Activity 3).
After completing its analysis, the EQRO writes a statement of findings about the MCP’s
information system. This statement should include implications of the findings for the following:
•

Completeness and accuracy of encounter data collected and submitted to the state

•

Validation and/or calculation of performance measures

•

Completeness and accuracy of tracking of grievances and appeals

•

Utility of the information system to conduct MCP quality assessment and improvement
initiatives

•

Ability of the information system to conduct MCP quality assessment and improvement
initiatives

•

Ability of the information system to oversee and manage the delivery of health care to the
MCP’s enrollees

•

Ability of the information system to generate complete, accurate, and timely T-MSIS data

•

Utility of the information system for review of provider network adequacy

•

Utility of the MCP’s information system for linking to other information sources for qualityrelated reporting (e.g., immunization registries, health information exchanges, state vital
statistics, public health data)

A.6 | APPENDIX A

Activity 5: Draft ISCA Summary for EQR Technical Report
A summary of the ISCA should be included in the EQR technical report developed by the
EQRO. This summary should include:
•

When the most recent ISCA was completed

•

The statement of the findings from the review

•

Overall findings from the review

•

Based on findings from the ISCA, recommendations to the state relevant to EQR-related
activities and/or revisions to the state’s managed care quality strategy

The Future of Information Systems Assessment
With increasingly sophisticated and comprehensive information systems, it is important to adapt
the way information systems’ capabilities are assessed. As information systems evolve, so will
the tools and rules with which states and EQROs assess them. As an example, information
systems may now be built on on-premise physical infrastructure, a cloud platform, or a hybrid of
both, which requires the ability to assess system security on these platforms to ensure the privacy
and security of protected health information (PHI) data. Supports for meeting existing statutory
requirements regarding privacy and security, including guidance and tools, might be considered
suitable topics for one of two optional EQR-related activities, additional performance
improvement projects (Protocol 8. Implementation of Additional Performance Improvement
Projects) or focus studies (Protocol 9. Conducting Focus Studies of Health Care quality).
Given the ongoing and accelerating accumulation of health information technology (HIT)
standards, HIT certification requirements, and HIT qualifications proposed and imposed by
federal payers, organizations should anticipate changes in assessments of new information
system requirements. Two recent developments are particularly pertinent to the future of
information systems assessment:
1. In 2011, CMS began working with state agencies and other stakeholders to finalize a new
data infrastructure, the Transformed Medicaid Statistical Information System (T-MSIS). TMSIS is designed to modernize the way that states submit data about beneficiaries, providers,
fee-for-service claims, and encounters to CMS. T-MSIS (1) expands required data elements
on person-level eligibility and services; (2) captures data on providers, managed care plans,
and third-party insurance; (3) provides for improved quality of state data; and (4) requires
states to submit data monthly instead of quarterly, making the data available sooner. CMS
expects that state agencies will thoroughly audit the managed care data to ensure that it
complies with all T-MSIS requirements before submission to CMS
2. New efforts to implement value-based purchasing, alternative payment models, and
integrated care models will require assessment of the MCPs’ ability to track (1) bundled,
incentive, bonus, and capitated payments, (2) whether all the needed services were delivered,
APPENDIX A | A.7

and (3) how clinical quality data for performance measurement is captured and
communicated back and forth to care managers. Including all paid amounts on encounter data
provides important information to the state and CMS, enabling more data-driven analytic
methods to value-based purchasing efforts and rate development
Of continuing importance to successful information systems is the adoption and meaningful use
of certified electronic health records (EHRs). The passage in 2016 of the 21st Century Cures
Act 90—which is designed to help improve care delivery by ensuring the interoperability of health
information exchange (HIE) systems for seamless patient care through increased coordination
and continuity of health care among health care providers—highlights the significance of
certified EHRs and HIE systems as potential drivers of improvements in individual and
population health. The design and utilization of secure EHRs will become an increasingly
important element in the EQR process as is reflected in the questions included in the ISCA tool.
States and MCPs should work collaboratively in the planning and use of certified EHRs and
health information exchange systems.
States and MCPs must also coordinate their HIT planning efforts to ensure interoperability
between systems that effectively provide for future data needs to meet eligibility, enrollment,
Health Insurance Exchange, quality reporting, and delivery system reform statutory and
regulatory requirements. EQROs should continually assess MCP planning activities to ensure
alignment with and responsiveness to these initiatives. For example, this could include use of
data from bi-directional data exchange with immunization registries to support state reporting of
the CMS Child Core Set immunization measures.
To learn from and share state experiences with emerging HIT and EHR initiatives that can
impact performance measure and performance improvement project outcomes reporting, CMS
strongly encourages states that contract with EQROs to include results of state HIT and EHR
initiatives impacted by MCP reporting in annual EQR reports. This may include successful
implementation of health information exchange with various state agencies to improve data
source collection efforts for performance measures (such as electronic clinical quality measures
or other administrative data sources used in the calculation of quality measures) or performance
improvement projects. Similarly, including lessons learned from challenging or unsuccessful
HIT initiatives are just as informative to federal and other state partners, and may also be
included in annual EQR technical reports.
END OF APPENDIX A

See https://www.gpo.gov/fdsys/pkg/PLAW-114publ255/content-detail.html for more information about the 21st Century
Cures Act.

A.8 | APPENDIX A

Worksheets for Appendix A
Worksheet A.1. Information System Capabilities Assessment (ISCA) Tool
Instructions. The ISCA tool is an information collection tool provided to an MCP 91 by the state or its EQRO to obtain
the information needed to validate the capabilities of the MCP’s information systems, processes, and data, with the
intent of supporting annual EQR-related activities and associated EQR analysis and recommendations as
documented in the EQR technical report. The state or its EQRO will define a timeframe in which the MCP is expected
to complete and return the tool or comparable information. For purposes of this worksheet, it is assumed the MCP will
record data in this tool. Documents from the MCP requested throughout the tool are listed in the Summary of
Requested Documentation Checklist, below.
The state and the MCP should be certain that data being reported are not only accurate today, but also have a reasonable
chance of being accurate for future reporting periods. Future accuracy can be predicted by assessing the MCP’s system
development cycle and supporting environment. Plans that lack development checkpoints and controls are much more likely
to introduce errors as systems change. The questions in this tool can be used to subjectively assess the likelihood of future
reporting anomalies. However, it should be noted that very few entities with information systems meet all the desirable
criteria. The EQRO is directed to consider the status of checkpoints and controls in its overall assessment of findings.
If the MCP’s information has been formally assessed within the last two years, please attach a copy of the
assessment report. Complete only those sections of the ISCA tool that were not covered by or have changed since
the formal assessment was conducted. If applicable, attach a copy of the MCP’s most recent information systems
analysis completed as a part of an accreditation review or third party performance measure validation process.
Note: The information requested in the ISCA pertains to the collection and processing of data for an MCP’s Medicaid
and/or CHIP line of business. In many situations, if not most, this may be no different than how an MCP collects and
processes commercial or Medicare data. However, for questions that address areas where Medicaid or CHIP data
are managed differently than commercial or other data, please provide the answers to the questions as they relate to
Medicaid or CHIP enrollees and Medicaid or CHIP data.
Any time there is a system difference between Medicaid and CHIP, it should be reported in the MCP’s responses.
However, unless noted, it is assumed that the MCP treats data from these two programs in the same manner.

MCP Contact Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, E-mail address, and date of interview, if applicable.
MCP Name:
MCP Contact Name:
Title:
Mailing address:
Phone number:
E-mail address:

For the purposes of the Appendix A worksheets, the term MCP includes managed care organizations (MCOs),
prepaid inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs), but does not include PCCM
entities, because 42 CFR Section 342, which is the basis for the requirement that states ensure maintenance of
health information systems, is only applicable to MCOs, PIHPs and PAHPs.

APPENDIX A | A.9

MCP Name:
Interview Date:
Type of delivery system
(check all that apply)

□ MCO □ PIHP □ PAHP
□ Other (specify): ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid and CHIP

Summary of Requested Documentation Checklist
Instructions. As you complete the ISCA tool and gather the files, please label all attached documentation as
described in the table column “Requested Document,” and when applicable by the activity number from the ISCA.
You are not limited to providing only the documentation listed below; you are encouraged to provide any additional
documentation that helps clarify an answer or eliminates the need for a lengthy response.
Check box if
document is
attached

Requested Document

Details

Previous Medicaid
Performance Measure Audit
Reports

If applicable, attach the information system analysis report
completed as a part of the MCP’s most recent accreditation
review or its most recent third party performance measure
validation process

Organizational Chart

Attach an organizational chart for your MCP. The chart should
make clear the relationship among key
Individuals/departments responsible for information
management, including performance measure reporting.

Data Integration Flow Chart

Attach a flowchart that gives an overview of the structure of
your management information system. See the example
provided in Section II-D. “Integration and Control of Data for
Performance Measure Reporting.” Be sure to show how all
claims, encounter, membership, provider, EHR, and vendor
data are integrated for performance measure reporting.

Performance Measure
Repository File Structure (if
applicable)

Attach a complete file structure, file format, and field definitions
for the performance measure repository.

Program/Query Language for
Performance Measure
Repository Reporting (if
applicable)

Attach full documentation on the software programs or codes
used to convert performance measure repository data to
performance measures.

Continuous Enrollment Source
Code

Attach a copy of the source code that you use to calculate
continuous enrollment for Medicaid or CHIP enrollees. If no
source code is use, then provide the computer program used

Medicaid Member Months
Source Code

Attach a copy of the source code/computer programs that you
use to calculate member months, member years for Medicaid
or CHIP enrollees.

Medicaid or CHIP Claims Edits

Attach a list of specific edits performed on claims as they are
adjudicated with notation of performance timing (pre- or postpayment) and whether they are manual or automated
functions.

A.10 | APPENDIX A

Check box if
document is
attached

Requested Document
Statistics on Medicaid or CHIP
claims/encounters and other
administrative data

Details
Attach documentation that explains statistics reported in the
ISCA.

Section 1. Background Information
1. Please select your Managed Care Model. Mark only one.
□ MCO
□ PIHP
□ PAHP
2. What year was the MCP incorporated? ______________
3. Enter your average unduplicated member enrollment for the last three years. For each column enter the reference
year.
Insurer

Year 1:

Year 2:

Year 3:

Privately insured
Medicare
Medicaid
CHIP
Other (specify)
4. Has your organization ever undergone a formal information system capability assessment?
□ Yes
□ No (GO TO SECTION 2)
4a. If yes, who performed the assessment? ______________
4b. When was the assessment completed? ______________
4c. Please provide a copy of the results of each assessment performed within the past 2 years.

APPENDIX A | A.11

Section 2. Information Systems: Data Processing Procedures & Personnel
These questions attempt to determine the stability and expertise of the information system department. Responses
can provide additional insight into the development cycle responses. Outsourcing means using non-employees to get
the work done, sometimes off-site, in which case project specification, management, coordination, and acceptance
become key success factors. Enter an educated guess if the turnover rate is unknown.
1. What type of system or repository does your organization use to store Medicaid and CHIP claims and encounter
data?
2. Is this data system or repository located onsite or located in the cloud?
□ Onsite (GO TO QUESTION 3)
□ In the cloud
2a. If in the cloud, which cloud provider hosts the data?
3. How would you characterize this system or repository? Mark all that apply.
□ Relational database management system (DBMS)
□ Network
□ Hierarchical DBMS
□ Flat file
□ Indexed
□ Proprietary
□ Other: Specify _____________
□ Don’t know
4. Into what repository or DBMS(s), if any, do you extract relevant Medicaid or CHIP encounter/claim/enrollment
detail for analytic reporting purposes?
5. How would you characterize the repository/DBMS(s)? Mark all that apply.
□ Relational DBMS
□ Network
□ Hierarchical DBMS
□ Flat file
□ Indexed
□ Proprietary
□ Other: Specify _____________
□ Don’t know
6. What programming language(s) do you use to create Medicaid and CHIP data extracts or analytic reports?
___________________________________________________________________
6a. How many staff are trained and capable of modifying these programs?
_______________
7. Do you calculate defect rates for programs?
□ Yes
□ No (GO TO QUESTION 8)

A.12 | APPENDIX A

7a. If yes, what methods do you use to calculate the defect rate?
7b. What was the most recent time period?
7c. What were the results?
8. Approximately what percentage of your organization’s programming work is outsourced?

_______%

9. What is the average years of experience among those staff who perform programing and data analysis in your
organization?
10. Approximately how many resources (time, money, etc.) are spent on training per programmer and analysis staff
per year?
Number of hours: ___________________
Dollars spent: $__________________
Other resources (specify): ___________________
10a. What type of training for programmers is provided?
11. What is the turnover rate for your programming and analysis staff for each of the last 3 years (new staff per
year/total staff)?
Year 1 (20xx): _____ % Year 2 (20xx): _____ % Year 3 (20xx): _____ %
12. Does your organization follow a standard software development methodology (SDLMC)?
□ Yes
□ No (GO TO QUESTION 13)
12a. Outline the steps of the maintenance cycle for your state’s mandated Medicaid and CHIP reporting
requirement(s). Include any tasks related to documentation, debugging, roll out, training, etc. The level of
detail should result in 10–25 steps in the outline.
13. Does your organization use version control software for change management and deployment to the production
environment?
□ Yes
□ No (GO TO QUESTION 14)
13a. If yes, what product is used?
Note (Q13a): The information system department should follow a standardized process when
updating and revising code. This process should include safeguards that ensure that the correct
version of a program is in use

13b. Do all programmer and analysis staff and all of your systems use this product for development and
deployment?
□ Yes
□ No

APPENDIX A | A.13

14. How does your organization know if changes to the claims/encounter/enrollment tracking system affect required
reporting to the state Medicaid or CHIP program (e.g., what prompts your organization to change these systems)?
Note (Q14): A specific individual within the organization should be responsible for determining the
impact of any changes made to the plan’s claims/encounter/enrollment tracking systems. The plan
should have in place a system for triggering information system staff to update the programs.

15. Who is responsible for your organization meeting the state Medicaid and CHIP reporting requirements? Mark all
that apply.
□ CEO
□ CFO
□ COO
□ CCO
□ Other (Specify) _____________________

Section 3. Staffing
1. Describe the Medicaid or CHIP data processing organization in terms of staffing and the expected productivity
goals. What is the overall daily, monthly, and annual productivity of the overall department and by processor?
Note (Q1): Unusually high productivity goals can affect the accuracy and quality of a processor’s
work.

2. Describe processor training from new hire to refresher courses for seasoned processors.
Note (Q2): New hires should be provided with on-the-job training and supervision. Supervisors
should closely audit the work of new hires before concluding the training process. Seasoned
processors should have occasional refresher courses and training concerning any system
modifications.

Section 4. Security
1. Does your organization have a disaster recovery (DR) plan and DR system?
□ Yes
□ No (GO TO QUESTION 6)
2. Where is the DR system located?
3. Does it provide failover capability?
4. How long does it take to switch over to the DR system when the primary system fails?
5. How often is the DR system tested?

A.14 | APPENDIX A

6. How frequently are system backups performed?
7. Where are backup data stored?
8. How and how often are the backups tested to make sure that the backup procedure is functioning properly?
9. How is Medicaid or CHIP data corruption prevented due to system failure or program error?
Note (Q9): A back-up procedure will protect the data from destruction due to system failure and
program error. Plans can also institute additional safeguards to protect data from being written
over during these processes.

10. Describe the controls used to assure that all Medicaid and CHIP claims data entered into the system are fully
accounted for (e.g., batch control sheets).
Note (Q10): The plan should have a process in place that ensures that all claims/encounters that
have been logged as received are entered into the system and processed.

11. Describe the provisions in place for physical security of the computer system and manual files:
Premises:
Documents:
Computer facilities:
Desktops, laptops and mobile devices:
Note (Q11): The system should be protected from both unauthorized usage and accidental
damage. Paper based claims/encounters should be in locked storage facilities when not in use.
The computer system and terminals should be protected from unauthorized access using a
password system and security screens. Passwords should be changed frequently and should be
re-set whenever an employee terminates

12. Describe the steps taken to verify that the MCP’s information system processes for protecting PHI, including its
encryption methods, are compliant with Federal Information Processing Standards Publication (FIPS) 140-2 (for
more information on the FIPS 140-2 process and validation list, please review the FIPS 140-2 related documents
at https://www.hhs.gov/sites/default/files/ocr/privacy/hipaa/administrative/securityrule/fips1402.pdf?language=es).
12a. Provide the results of the most recent FIPS 140-2 tests completed on the MCP’s information system.
13. Describe the procedures in place to determine which system users may access levels of the system that include
PII. Please identify the job titles and responsibilities of each system user with access to systems that include PII.
14. Describe the methods in place to allow those with access to PII to only access the minimum amount of
information necessary to perform their job.

APPENDIX A | A.15

15. Identify training and awareness provided to personnel (system owners, managers, operators, contractors and/or
program managers) using the system to make them aware of their responsibilities for protecting the information
being collected and maintained.
16. Describe the process and guidelines in place with regard to the retention and destruction of PII.
17. Describe, briefly but with specificity, how the PII will be secured in the system using administrative, technical, and
physical controls.
18. If you employ cloud-based technology, describe the provisions in place to secure the virtual system.
19. If you utilize remote network access to connect users with the MCP’s secure networks via the internet, describe
the provisions in place to secure the network against unauthorized access.
20. Which staff position(s) is responsible for the security and user administration task that grants access to the
system?
21. Which staff positions have access to what levels of the system?
22. Can your programming and analysis staff access the production system or only the development system?
□ Production system only
□ Development system only
23. How often must passwords be changed?
24. How quickly are logons deactivated after employee terminations and resignations?
25. Describe your patch management protocols and processes.
26. What other individuals have access to the computer system? Customers? Providers? Describe their access and
the security that is maintained restricting or controlling such access.
Note (Q26): Both members and providers should have their access limited to read-only so that
they cannot alter any files. They should be given access to only those files containing their own
patients or members. Customers should be prevented from accessing highly confidential patient
information by being given “blinded” patient names and “scrambled” ID numbers, or restricted
access to particular files.

Section 5. Data Acquisition Capabilities
The purpose of this section is to obtain a high-level understanding of how you collect and maintain administrative
data (claims and encounter data), enrollment information, data on ancillary services such as prescription drugs.
A. Administrative Data (Claims and Encounter Data)
These questions request information on input data sources (e.g., electronic claims and paper) and on the transaction
system(s) you use.

A.16 | APPENDIX A

1. How are data submitted (e.g. electronically, on paper or both)?
□ Submitted electronically
□ Submitted on paper
□ Submitted both electronically and on paper
1a. What percent of data are submitted electronically?
1b. What formats are used?
1c. Is there a front-end web portal available for data submissions?
□ Yes
□ No
2. Do you use standard claims or encounter forms for the following? Mark yes or no for each data source. If yes,
please specify (e.g., CMS1500, UB 94).
Data source

Yes

If “Yes,” please specify

Hospital
Physician
Drug
Nursing home
Home health
Mental health
Dental

Note (Q2): Plans that do not use either CMS 1500 or UB 92 forms may be using forms they
developed themselves. If a plan is using its own forms, these forms should be reviewed to ensure
they are capturing the following key data elements: patient identification information (Medicaid
ID, name, date of birth, gender), provider identifying information (national provider identifier (NPI),
Tax ID, name), date of service, place of service and diagnoses and procedure codes. An
evaluation of their forms to ascertain adequacy and completeness of data collection may be
necessary.

3. We would like to understand how claims or encounters are submitted to your plan. We are also interested in an
estimate on an annual basis of what percentage (if any) of services provided to your enrollees by all providers
serving your Medicaid and CHIP enrollees are NOT submitted as claims or encounters, and therefore, are not
represented in your administrative data. Please fill in the following table with the appropriate percentages:
Claims or Encounter Types
Medium

Hospital

PCP

Specialist
Physician

Dental

Mental health/
substance abuse

Drug

Other

Claims/encounters
submitted electronically
Claims/encounters
submitted on paper

APPENDIX A | A.17

Claims or Encounter Types
Medium

Hospital

PCP

Specialist
Physician

Dental

Mental health/
substance abuse

Drug

Other

Services not submitted
as claims or encounters
TOTAL

100%

Note (Q3): Since paper forms need to be entered into a plan’s system, processing paper forms is
prone to error. If a plan is receiving more that 50 percent of its data on paper forms, verify the
data checks the plan uses to test processor accuracy. Electronic data submission should also
undergo data edits and validity checks. Plans with a high percentage of unavailable data for a
particular category will have difficulty reporting measures that use that category. For example, a
plan receiving no drug data from its vendor would not be able to report the HEDIS® measures for
Outpatient Drug Utilization.

3a. For each type of claims or encounter type for which some percentage are not represented in your
administrative data, please explain why such activity is not reported.
4. In the following table, please enter an “R” in the appropriate cell if the following data elements (data fields) are
required by you for providers, for each of the types of Medicaid claims/encounters identified below. Note that each
of these elements is required by T-MSIS, and that the MCP’s data elements should align with T-MSIS
requirements:
Claims or Encounter Types
Medium
Patient

gendera

Patient date of birth
and age
ICD9/10 Diagnosis
Codes
Procedure Code
Types:
CPT-4/HCPCS
National Drug Code
(NDC)
Universal Product
Code (UPC
Manufacturer Part
Number (MPN)
First date of service
Last date of service
Quantity of Service
Revenue code

A.18 | APPENDIX A

Hospital

PCP

Specialist
Physician

Dental

Mental health/
substance abuse

Drug

Other

Claims or Encounter Types
Medium

Hospital

Specialist
Physician

PCP

Dental

Mental health/
substance abuse

Drug

Other

Provider NPI
Provider specialty
TOTAL

100%

a
UN, M, or F (UN = the gender of a person could not be uniquely defined as male or female; M = Male; F = Female). Please see
AHRQ’s Administrative Gender Value Set document at https://ushik.ahrq.gov/ViewItemDetails?&system=mu&itemKey=86667000
for more information.

Note (Q4): Standard measures of plan performance such as Medicaid HEDIS® are dependent upon
the availability of the fields listed above. If procedure codes or diagnosis codes are not available,
the data will not include the necessary level of detail to report performance measures.

5. In the following table, please enter how many diagnoses and procedures are captured on each claim and on each
encounter:
Claim
Diagnoses

Encounter
Procedures

Diagnoses

Procedures

Institutional data
Provider/Provider group data

Note (Q5): All diagnosis codes types should be standard, nationally recognized codes, rather than
plan-specific codes. Diagnosis code fields should include all diagnosis codes needed to identify
the reason for the encounter, and all relevant comorbidities and complications should be
included. Each service rendered or product dispensed should be identified with the appropriate
identifier.

6. Can you distinguish between principal and secondary diagnoses?
□ Yes
□ No
Note (Q6): Some plans will consider the first diagnosis on the claim to be principal. Other plans
determine the principal diagnosis by selecting the most expensive condition represented.

6a. If “Yes” to 6, above, how do you distinguish between principal and secondary diagnoses?
7. Please explain what happens if a Medicaid or CHIP claim or encounter is submitted and one or more required
fields are missing, incomplete, or invalid. For example, if diagnosis is not coded, is the claims examiner required
by the system to use an on-line software product like AutoCoder to determine the correct ICD-10 code?
Institutional Data:
Professional Data:

APPENDIX A | A.19

8. How is the MCP able to distinguish backend-system-assigned data versus data submitted by the service
provider?
9. What steps do you take to verify the accuracy of submitted information (e.g., procedure code, diagnosis edits,
gender-diagnosis edits, gender-procedure code edits)?
Institutional Data:
Professional Data:
Note (Q9): Plans will often verify that the information in procedure code and diagnosis code fields
are valid codes. Plans may also verify that diagnosis and procedure codes are appropriate for age
and gender. For example, a claim with a procedure of hysterectomy should be for a female
patient.

10. Under what circumstances can claims processors change Medicaid or CHIP claims/encounter information?
Note (Q10): If processors are given the ability to modify claims/encounter information, the
accuracy of that information could be affected either negatively or positively. Processors may
simply correct data that was submitted incorrectly, which would increase the quality of the data.
However, processors may also change diagnosis and procedure codes which could result in a loss
of coding specificity. Does the plan check processed data against paper claims?

11. Identify any instance where the content of a field is intentionally different from the description or intended use of
the field. For example, if the dependent’s SSN is unknown, do you enter the member’s SSN instead?
Note (Q11): Changing the content of a field can create data processing issues. For example, if the
enrollee’s SSN is used as an ID for a number of dependents, the claim may be given the age and
sex of the member rather than the actual patient. The use of the enrollee’s SSN would make it
difficult to track the dependent’s experience over time.

12. How are Medicaid or CHIP claims/encounters received from each of the following sources? Please mark one
column per source:
Source

Received directly from provider

Received through an intermediary

Hospital
Physician
Pharmacy
Nursing home
Home health
Mental health
Dental
Other
12a. If the data are received through an intermediary, what changes, if any, are made to the data? Please answer
for each source received through an intermediary in the table above.

A.20 | APPENDIX A

Note (Q12): Intermediaries that are processing the data, such as a pharmacy benefit firm, could
modify the data, creating a data set that is inconsistent with the plan’s data. The intermediary
may define field content differently or may not be using the same fields as the plan, making it
difficult to integrate the intermediary’s data into the plan’s systems. All data submitted through an
intermediary should be monitored for quality by the plan.

13. In the following table, please estimate the percentage of Medicaid or CHIP claims/encounters that are coded
using the following coding schemes:
Coding scheme

Inpatient
diagnosis

Inpatient
procedure

Ambulatory/outpatient Ambulatory/outpat
diagnosis
ient procedure

Drug

ICD-10 CM
CPT®-4
HCPCS
DSM-IV
National Drug Code
Internally developed
Other: Specify
Not required
TOTAL (can be
greater than 100% if
a claims type is
subject to more than
one coding system)

Note (Q13): If a plan is using internally-developed coding schemes, the state should verify
whether this coding can be mapped to standard coding such as ICD-10 or CPT-4. If the coding can
be translated for reporting purposes (Medicaid HEDIS® requires diagnosis and procedure codes),
the plan should provide information on the level of specificity with which the coding maps to
standard coding (e.g., three-digit specificity or five-digit specificity). If the mapping has a low
level of specificity, information on co-morbidities and complications may not be retained during
translation.

14. Please list all information systems through which service and utilization data for the Medicaid or CHIP population
is processed.
15. Please describe any major systems changes or updates that have taken place in the last three years in your
Medicaid or CHIP claims or encounter system (be sure to provide specific dates on which changes were
implemented). Check all that apply
□ New system installed to replace old system
□ New system purchased and installed to replace most of old system; old system still used
- Major enhancements to old system

APPENDIX A | A.21

□ If enhancements were made to the old system, please summarize what enhancements were made and
whether (and if so, how) the enhancements have impacted historical data:
□ New product line adjudicated on old system
□ Conversion of a product line from one system to another
Note (Q15): When a plan undertakes any major system changes such as an upgrade or conversion
to a new system, the system changes could affect data quality. Data quality problems include
corruption of data, loss of data, and loss of the level of detail within the data. The implementation
of a new system can also affect access to historical data. Changes in data quality and access will
affect the plan’s ability to report performance measures and utilization. The plan should have a
fallback option, such as parallel operations.

16. How many years of Medicaid or CHIP data are retained on-line?
16a. How is historical Medicaid or CHIP data accessed when needed?
Note (Q16a): Due to system constraints, a plan may remove historical data and place it in off-line
storage. The MCP’s ability to report on experience spanning several years of data could be
affected by the accessibility of the data stored off-line.

17. What percent of your Medicaid or CHIP data is processed on-line vs. batch? If batch, how often are batch jobs
run?
18. Describe your policy regarding Medicaid or CHIP claim/encounter audits.
18a. Are Medicaid or CHIP encounters audited regularly or randomly?
□ Regularly
□ Randomly
18b. What are the standards regarding timeliness of processing?
Note (Q18b): Plans should be performing random periodic audits of their encounter data to
determine the quality of data processing. Plans that do not perform audits at least annually are
not closely monitoring the quality of data processing. Plan standards regarding timeliness of
processing will influence the lag time for encounter data processing.

19. Please describe system edits that are targeted to field content and consistency. Are diagnostic and procedure
codes edited for validity?
Note (Q19): MCPs should have an established, standard set of edits that verify field content and
consistency. For example, a field content data edit would verify that a valid date is entered into
the date of service field. Key fields which should be edited include patient identifying information
(Medicaid ID, name, date of birth, sex), provider identifying information (name, tax ID, type), date
and place of service, and diagnosis and procedure codes. The quality of diagnosis and procedure
coding will affect the validity of reports and performance measures submitted by the MCP/PIHP.

A.22 | APPENDIX A

20. Please complete the following table for Medicaid and CHIP claims and encounter data and other Medicaid and
CHIP administrative data. Attach any documentation that should be reviewed to explain the data that is being
submitted.
Item

Claims

Encounters

Other administrative data

Percent of total service volume
Percent complete
How are the above statistics
quantified?
Incentives for data submission

Note (Q20): MCPs with claims data comprising more than 50 percent of their total service volume
are likely to have a more complete representation of total MCP experience than MCPs that rely
heavily on encounter data. While providers have an incentive to submit claims in order to receive
payment for services, they do not always have incentives to submit encounter information. If an
MCP does not offer providers an incentive, or does not require the submission of encounter data,
the MCP may not receive data for every encounter. Other administrative data collected by an MCP
could include data from pharmacy or laboratory vendors.

21. Describe the Medicaid or CHIP claims/encounter suspend (“pend”) process including timeliness of reconciling
pended services. What percentage of claims are suspended or pended?
Note (Q21): Pended claims/encounters are those claims/encounters that have been suspended
during processing because they failed data quality edits or violated provider payment parameters.
Information on these claims and encounters will not be available for reporting until they have
been reconciled and processed into the system.

22. Describe how Medicaid or CHIP claims are suspended/pended for medical review, for non-approval due to
missing authorization code(s) or for other reasons. What triggers a processor to follow up on “pended” claims?
How frequent are these triggers?
Note (Q22): Review and processing should not be handled by the same employee. A system should
be in place which encourages the processor to follow-up on the status of claims in review that
have not yet been approved to ensure they are resolved.

23. Are any of your Medicaid or CHIP services/providers capitated?
23a. If yes, have you conducted studies on the completeness of the information collected on capitated services?
23b. If yes, what were the results?

APPENDIX A | A.23

Note (Q23b): Because provider payment for capitated services is not determined by the encounter
data submitted, providers do not have an incentive to submit complete and accurate information
on every service provided. Data on capitated services often does not include the same level of
detail as fee-for-service claims information. Per service pricing information may not be available
when providers are paid on a capitated basis but at least the amount of the capitation payment
should be available. Plans should be aware that capitated data is less complete and should audit
the data at least annually to monitor its quality.

24. In the following table, enter the claim/encounter system(s) for each product line offered to Medicaid or CHIP
enrollees.
Note (Q24): Typically, there is just one product line offered to Medicaid or CHIP enrollees, but
there may be some circumstances in which an MCP offers additional product lines to the state
(e.g., partial risk products, premium assistance programs).

Medicaid
Systems Used to Process

Product Line:

____________________

____________________ ____________________

Product Line:
____________________

Fee-for-service (indemnity) claims
Capitated service encounters
Clinic patient registrations
Pharmacy claims
Other (describe)

CHIP (if applicable)
Systems Used to Process

Product Line:

____________________

Fee-for-service (indemnity)
claims
Capitated service encounters
Clinic patient registrations
Pharmacy claims
Other (describe)
25. Beginning with receipt of a Medicaid or CHIP claim in-house, describe the claim handling, logging, and processes
that precede adjudication. Describe the following: When are claims assigned a document control number and
logged or scanned into the system? When are claims stored using document imaging? If there is a delay in
document imaging, how do processors access a claim that is logged into the system, but is not yet filmed?
25a. Please describe each system or process that is involved in adjudicating:

A.24 | APPENDIX A

A professional encounter(s) for a capitated service (e.g., childhood immunizations that arrive separately
from the office visit)

A hospital claim for a delivery or for a newborn that exceeds its mother’s stay

Note (Q25a): Professional encounters arriving separately from an office visit may not be
processed as quickly as the actual office visits. If these encounters are treated as “non-standard”
events, the plan may not be able to easily link these encounters with the related office visit. For
example, newborns exceeding a mother’s stay may have their hospital stay split into two parts.
The part of the stay which coincides with the mother’s hospitalization may be processed on the
mother’s claim and the remainder of the stay could be processed separately. Processing the
newborn’s stay as two separate claims could affect the plan’s ability to report accurately on
newborn hospital utilization.

25b. Discuss which decisions in processing a Medicaid or CHIP claim/encounter are automated, which are
prompted by automated messages appearing on the screen, and which are manual. Document the
opportunities a processor has for overriding the system manually. Is there a report documenting overrides or
“exceptions” generated on each processor and reviewed by the claim supervisor? If so, please describe this
report.
25c. Are any outside parties or contractors used to complete adjudication, including but not limited to:
-

Bill auditors (hospital claims, claims over a certain dollar amount)
□ Yes
□ No

Peer or medical reviewers
□ Yes
□ No

Sources for additional charge data (usual & customary)
□ Yes
□ No

Bill “re-pricing” for carved out benefits (mental health, substance abuse)
□ Yes
□ No

Other
□ Yes (If yes, please provide additional information)
□ No

25d. How are these data incorporated into your organization’s data?
Note (Q25d): If outside parties are used, the plan should be incorporating data generated by those
parties into the system. The data should first be run through the plan’s data quality checks to
verify its accuracy and completeness.

25e. Describe the system’s editing capabilities that assure that Medicaid and CHIP claims are correctly
adjudicated

APPENDIX A | A.25

Attach a list of the specific edits that are performed on claims as they are adjudicated, and note (1)
whether the edits are performed pre- or post-payment, and (2) which are manual functions and which are
automated functions.

Note (Q25e): When reviewing plan adjudication edits, the state should concentrate on edits which
affect the data fields that are used to generate plan performance measures and reports. Are
outliers for length of stay and charges edited? Utilizing an automated editing process provides
more consistent results that do not require processor judgment. Edits that are performed prepayment can prevent invalid data from being incorporated into the system.

25f. Discuss the routine and non-routine (ad hoc or special) audits that are performed on claims/encounters to
assure the quality and accuracy and timeliness of processing. In your response, note which audits are
performed per processor, which rely on targeted samples, and which use random sampling techniques. What
is the total percentage of claims on-hand that are audited through these QA processes? How frequently do
these audits occur?
Note (Q25f): This item is not relevant in instances where the EQRO is performing encounter data
validation. When reviewing edits that are used to determine processor accuracy, consider that
these edits will not provide information on the quality of the initial provider data submission. The
audit plan should include random sampling techniques to provide an overall picture of quality.
Plans will often concentrate on auditing complicated or aberrant claims/encounters rather than
using a random sample. The plan should have instituted a process for sharing audit results with
the processor to facilitate quality improvement.

25g. Please describe how Medicaid and CHIP eligibility files are updated, how frequently and who has “change”
authority. How and when does Medicaid and CHIP eligibility verification take place?
25h. How are encounters for capitated services handled by payment functions? What message appears to notify
processors that they are handling a capitated service?
25i. Describe how your systems and procedures handle validation and payment of Medicaid claims when
procedure codes are not provided.
Note (Q25i): Plans requiring valid procedure coding for all claims/encounters will have more
detailed data available for reporting and analysis. However, these plans may allow processors to
supply missing codes using a code book or override the system using an unspecified code. A
number of plans use programs such as the GMIS AutoCoder product to fill in missing codes. When
a plan supplies missing codes, the coding can be less accurate than codes supplied directly by
the provider of service.

26. Describe all performance monitoring standards for Medicaid and CHIP claims/encounters processing. Provide the
results of a recent performance monitoring activity.
26a. How is performance against targets figured into the official performance appraisal process? Into processor
and supervisor compensation?

B. Enrollment System
1. Please describe any major changes/updates that have taken place in the last three years in your Medicaid or
CHIP enrollment data system. Include the specific dates on which changes were implemented. For example:

A.26 | APPENDIX A

New enrollment system purchased and installed to replace old system

New enrollment system purchased and installed to replace most of old system; is the old system still used?

Major enhancements to old system; what kinds of enhancements, and what impact on your historical data?

New product line members stored on old system
Note (Q1): Changes to a plan’s enrollment system requiring data conversion and data integration
can create data quality problems. Implementing a new enrollment system could lead to a loss of
access to data on the old system, or the assignment of new member numbers for all enrollees.
Data conversion and integration can also limit a plan’s ability to track an enrollee’s enrollment
history. When a new product line is added to an existing system, a plan may need to make the
new data fit the older process, therefore modifying the system to “handle” new information.
Implementing such modifications can be difficult for a plan that has been using the same system
for a number of years. The level of enrollment detail retained can be affected by such
modifications.

2. In your opinion, have any of these changes influenced, even temporarily, the quality and/or completeness of the
Medicaid or CHIP data that are collected? If so, how and when?
Note (Q2): Consider whether changes in data quality will affect the validity of the data submitted
to the state.

3. How does your plan uniquely identify enrollees?
Note (Q3): Major changes to an MCP’s enrollment system could involve the conversion of
membership data to a new system. When MCPs convert members, they may change the enrollee’s
ID number, making it difficult to track the enrollee’s enrollment pattern across time. Changes to
the enrollment system could also lead to a loss of data for specific patients.

4. How do you handle enrollee disenrollment and re-enrollment in the Medicaid or CHIP product line? Does the
member retain the same ID?
Note (Q4): Enrollees should have a single ID number to facilitate tracking their experience.
However, some plans change an enrollee’s ID number when the enrollee re-enrolls. Experience for
enrollees who have switched ID numbers will be more difficult to track. Dependents using an
enrollee’s ID are also difficult to identify for reporting purposes. For example, children without a
unique ID could affect the ability of the plan to report on low birth-weight babies, childhood
immunizations, and asthma inpatient admissions. This is an important point. EQROs should give
higher “grades” to plans that use strong methods of identifying enrollees.

5. Can your systems track enrollees who switch from one product line (e.g., Medicaid, commercial plan, Medicare)
to another?
□ Yes
□ No
5a. Can you track an enrollee’s initial enrollment date with your MCP?

APPENDIX A | A.27

□ Yes (GO TO QUESTION 5C)
□ No
5b. If not, is a new enrollment date assigned when a member enrolls in a new product line?
□ Yes
□ No
5c. Can you track and link previous claim/encounter data across product lines?
□ Yes
□ No
6. Under what circumstances, if any, can a Medicaid or CHIP member exist under more than one identification
number within your MCP’s information management systems? Under what circumstances, if any, can a member’s
identification number change?
7. How does your MCP enroll and track newborns born to an existing Medicaid or CHIP enrollee?
7a. If your MCP has a Medicare product line, describe how your enrollment systems link individuals
simultaneously enrolled in both your Medicare product line and the Medicaid plan product line.
8. Is claim/encounter data linked for Medicare/Medicaid dual eligibles so that all encounter data can be identified for
the purposes of performance measure reporting?
□ Yes
□ No
8a. Is claim/encounter data linked for individuals enrolled in both a Medicare and Medicaid plan so that all
encounter data can be identified for the purposes of performance measure reporting?
□ Yes
□ No
9. How often is Medicaid and CHIP enrollment information updated?
Note (Q9): Enrollment information should be updated real-time, daily, or weekly.

10. How is Medicaid and CHIP continuous enrollment being defined? In particular, does your system have any
limitations that preclude you from fully implementing continuous enrollment requirements exactly as specified in
the state performance measure requirements?
11. Please attach a copy of the source code that you use to calculate Medicaid/ CHIP continuous enrollment.
12. How do you handle breaks in Medicaid or CHIP enrollment, e.g., situations where a Medicaid enrollee is
disenrolled one day and re-enrolled the next simply for administrative reasons? Does this affect your continuous
enrollment calculations?
13. Do you have restrictions on when Medicaid or CHIP enrollees can enroll or disenroll? Please describe.
14. How do you identify and count the following:

A.28 | APPENDIX A

Medicaid member months?
Medicaid member years?
15. Please list all data from which claims/encounters for the Medicaid or CHIP product line are verified.
Note (Q15): Eligibility of the patient should be verified before claims and encounters are
processed. Dates of enrollment and disenrollment are key reporting fields for Medicaid HEDIS®
measures. Eligibility status is dynamic for Medicaid beneficiaries and should be updated
frequently. Eligibility status should also be verified before data is submitted to the state.

16. Does the plan offer vision or pharmacy benefits to its Medicaid or CHIP members that are different from the vision
or pharmacy benefits offered to its commercial enrollees (within a given contract or market area)?
□ Yes
□ No (GO TO SECTION C, ANCILLARY SYSTEMS)
16a. If vision benefits vary by benefit package, outline the different options available. How are enrollees
tracked?
16b. If pharmacy benefits vary by benefit package, outline the different options available. How are enrollees
tracked?
C. Ancillary Systems
Use this section to record information on stand-alone systems or benefits provided through subcontracts, such as
pharmacy or mental health/substance abuse.
1. Does your MCP incorporate data from one or more third-parties to calculate any of the following Medicaid and
CHIP quality measures? If so, which measures require third-party data?
Note (Q1): The measures listed in the following table are examples of measures that can be
calculated with administrative data and align with CMS quality measurement initiatives as of
2021. The state and EQRO should tailor this table to list the measures that the state requires its
MCP contractors to produce and any other measures in which the state is interested.

Measure

Third-Party Data Source

Well-Child Visits in the First 30 Months of Life (W30-CH)
Child and Adolescent Well-Care Visits (WCV-CH)
Childhood Immunization Status (IMA-CH)
Immunizations for Adolescents (IMA-CH)
Developmental Screening in the First Three Years of Life (DEV-CH)
Chlamydia Screening in Women Ages 16–20 (CHL-CH) and Ages
21 to 24 (CHL-AD)
Breast Cancer Screening (BCS-AD)
Cervical Cancer Screening (CCS-AD)

APPENDIX A | A.29

Measure

Third-Party Data Source

Prenatal and Postpartum Care: Timeliness of Prenatal Care (PPCCH) and Postpartum Care (PPC-AD)
Screening for Depression and Follow-Up Plan: Ages 12 to 17 (CDFCH) and Age 18 and Older (CDF-AD)
Follow-Up Care for Children Prescribed AttentionDeficit/Hyperactivity Disorder (ADHD) Medication (ADD-CH)
Follow-Up After Hospitalization for Mental Illness: Ages 6–20 (FUHCH) and Age 18 and Older (FUH-AD)
Initiation and Engagement of Alcohol and Other Drug Abuse or
Dependence Treatment (IET-AD)
Use of Opioids at High Dosage in Persons Without Cancer (OHDAD)
2. Describe any concerns you may have about the quality or completeness of any third-party data.
Note (Q2): If a plan is using third-party data, the plan should have a formal process in place to
validate that data before incorporating it into their information system. The plan needs to check
the third-party data for reliability, completeness and timeliness of submission.

3. Please list subcontracted Medicaid or CHIP benefits that are adjudicated through a separate system that belongs
to a third-party.
Note (Q3): Many plans contract out services for pharmacy benefits management, mental
health/substance abuse, laboratory and radiology services. If the data are processed on the thirdparty’s system, it may not be forwarded to the plan in a complete form or on a timely basis. Such
entities may also use a different method of processing resulting in data that will not merge with
or complement plan data.

4. Describe the kinds of information sources available to the MCP from the vendor (e.g., monthly hard copy reports,
full claims data).
5. Do you evaluate the quality of this information?
□ Yes
□ No (GO TO QUESTION 6a)
5a. If yes, how?
Note (Q5a): All of the third-party information should be verified for accuracy before a plan loads it
into their information system. The plan and the third-party data source may not define variables
consistently or use the same reporting format.

6. Did you incorporate these vendor data into the creation of Medicaid or CHIP-related studies?
□ Yes (GO TO SECTION D)

A.30 | APPENDIX A

□ No
6a. If no, why?
D. Additional Data Sources that Support Quality Reporting
This section requests any data sources beyond third party collection of claim/encounter data that support quality
reporting.
1. Does the MCP use any other data sources beyond claim/encounter data (such as, beneficiary provided data, HIE,
registry data source, vital statistics, etc.)?
□ Yes
□ No
If yes, please list additional data sources: _______________________________________________________
If yes, please describe how the MCP verifies the accuracy of the data and data exchange process for each data
source listed above.
E. Integration and Control of Data for Performance Measure Reporting
This section requests information on how your MCP integrates Medicaid and CHIP claims, encounter, membership,
provider, third-party, and other data to calculate performance rates. All questions relate to your current systems and
processes, unless indicated otherwise.
1. Please attach a flowchart outlining the structure of your management information systems, indicating data
integration (i.e., claims files, encounter files, etc.) at the most granular level you have it.
2. In consolidating data for Medicaid and CHIP performance measurement, how are the data sets for each measure
collected:
-

By querying the processing system online?

By using extract files created for analytical purposes? If so, how frequently are the files updated? How do they
account for claim and encounter submission and processing lags? How is the file creation process checked
for accuracy?

By using a separate relational database or data warehouse (i.e., a performance measure repository)? If so, is
this the same system from which all other reporting is produced?

3. Describe the procedure for consolidating Medicaid or CHIP claims/encounter, member, and provider data for
performance measure reporting (whether it is into a relational database or file extracts on a measure-by-measure
basis).
3a. How many different sources of data are merged together to create reports?
3b. What control processes are in place to ensure that data merges are accurate and complete?
3c. What control processes are in place to ensure that no extraneous data are captured (e.g., lack of specificity in
patient identifiers may lead to inclusion of non-eligible members or to double counting)?
4. Describe both the files accessed to create Medicaid or CHIP performance measures and the fields from those
files used for linking or analysis. Use either a schematic or text to respond.

APPENDIX A | A.31

5. Are any algorithms used to check the reasonableness of data integrated to report Medicaid or CHIP performance
measures?
6. Are Medicaid or CHIP reports created from a third-party software product?
□ Yes
□ No (GO TO QUESTION 7)
6a. If yes, how frequently are the files updated? How are reports checked for accuracy?
7. Are the data files used to report Medicaid or CHIP performance measures archived and labeled with the
performance period in question?
□ Yes
□ No
8. Information on several types of external encounter sources is requested. In the following table, please indicate the
following for each type of delegated service:
-

Column 2. Indicate the number of third-parties contracted (or subcontracted) to provide the Medicaid or CHIP
service. Count the entities that offer all or some of the portion of the service indicated.

Column 3. Indicate whether your MCP receives member-level data for any Medicaid or CHIP performance
measure reporting from the vendor(s). Only answer “Yes” if all data received from contracted third-parties(s)
are at the member level. If any encounter-related data is received in aggregate form, you should answer “No”.
If type of service is not a covered benefit, indicate “N/A”.

Column 4. Indicate whether all data needed for Medicaid or CHIP performance measure reporting are
integrated, at the member-level, with MCP administrative data.

Columns 5 and 6. Rank the completeness and quality of the Medicaid or CHIP data provided by the third
party(s). Consider data received from all sources when using the following data quality grades:
A. Data are complete or of high quality
B. Data are generally complete or of good quality
C. Data are incomplete or of poor quality

Column 7. Describe any concerns you have in ensuring completeness and quality of Medicaid or CHIP data
received from contracted third-parties. If the measure is not being calculated because there are no eligible
members, please indicate “N/A”.

Medicaid or CHIP Claim/Encounter Data from Third Parties

Type of
delegated
service

Number of
contracted
third-parties

Behavioral
health
Family planning
Home health
care

A.32 | APPENDIX A

Always receive
member-level
data from all
third party(s)
(Y or N)

Integrate thirdparty data with
MCP
administrative
data? (Y or N)

Data
completeness
(A, B, or C)

Data
quality
(A, B, or
C)

Describe
rating
concerns
with data
collection

Medicaid or CHIP Claim/Encounter Data from Third Parties

Type of
delegated
service

Number of
contracted
third-parties

Always receive
member-level
data from all
third party(s)
(Y or N)

Integrate thirdparty data with
MCP
administrative
data? (Y or N)

Data
completeness
(A, B, or C)

Data
quality
(A, B, or
C)

Describe
rating
concerns
with data
collection

Hospital
Laboratory
Pharmacy
Primary care
Radiology
Specialty care
Vision care
Dental for
children
9. Does your MCP use a performance measure repository?
□ Yes
□ No (GO TO QUESTION 10)
9a. If your MCP uses a performance measure repository for Medicaid or CHIP performance measures, review
the repository structure. Does it contain all the key information necessary for Medicaid or CHIP
performance measure reporting?
10. Please describe your Medicaid or CHIP report production logs and run controls.
10a. Please describe your Medicaid or CHIP performance measure report generation process.
11. How are Medicaid or CHIP report generation programs documented?
12. How does your MCP test the process used to create Medicaid and CHIP performance measure reports?
13. Are Medicaid and CHIP performance measure reporting programs reviewed by supervisory staff?
□ Yes
□ No
14. The purpose of these questions is to evaluate the Medicaid and CHIP provider compensation structure and
reporting of certain types of compensation, as this may influence the quality and completeness of data. Please
identify the percentage of member months in your plan contributed by Medicaid members whose primary care
providers and specialists are compensated through each of the following payment mechanisms:
Payment mechanism

Primary care physician

Specialist physician

Salaried
Fee-for-Service, no withhold or bonus

APPENDIX A | A.33

Payment mechanism

Primary care physician

Specialist physician

100%

Fee-for-Service, with withhold
Please specify % withhold:___________
Fee-for-Service with bonus
Bonus range: _____________________
Capitated - no withhold or bonus
Capitated with withhold
Please specify % withhold:___________
Capitated with bonus
Bonus range: _____________________
Global/bundled payments
Other: (Specify)___________________
TOTAL

Note (Q14): Timeliness and completeness of provider data submissions often varies by
contracting arrangement. Salaried providers work directly for the MCP and will submit data on a
timely basis if data submission is a parameter in their contract with the MCP. Fee-for-service
providers have the largest incentive to submit accurate and complete data since their payment
depends upon it. Capitated providers will need incentives to submit accurate and complete data.
Their compensation should be linked to data submission, which can be done through the use of
bonuses and withholds. For example, lag times may differ by compensation arrangement as
follows: Capitation/Salaried-no lag, Fee-for-Service - 60 day lag, Hospital - 45 day lag.

15. How are bonuses and penalties captured within your system? Is this information part of your standardized
reporting?
15a. Is the underlying data that determines whether and the extent of bonuses and penalties captured in your
system? Is this information part of your standard reporting?
15b. For bundled/global payments, how does your system capture information about the individual services
provided for this bundled/global payment? Is this information part of your standardized reporting?
15c. Does your system capture clinical data for quality measurement purposes for providers who receive
bundled/global payments? Is this information part of your standardized reporting?
16. Please describe how Medicaid or provider directories are updated, how frequently, and who has “change”
authority.
16a. Does your MCP maintain provider profiles on its website?
□ Yes
□ No (GO TO QUESTION 17)
16b. If yes to “16a,” what provider information is maintained in on the website (e.g., languages spoken, special
accessibility for individuals with special health care needs). Other? Please describe:

A.34 | APPENDIX A

17. Does your MCP maintain provider profiles on its information system?
□ Yes
□ No (GO TO QUESTION 18)
Note (Q17): Provider directories should be updated to reflect changes in provider status to
prevent members from selecting providers no longer under contract with the plan. The plan should
have adequate security procedures in place to restrict the number of individuals who can access
confidential provider information and institute changes in status.

17a. If yes to “17,” what provider information is maintained in the provider profile database (e.g., languages
spoken, special accessibility for individuals with special health care needs). Other? Please describe.
18. How are Medicaid or CHIP fee schedules and provider compensation rules maintained? Who has updating
authority?
Note (Q18): Since providers consider fee schedule and compensation information to be
confidential, access to this information should be restricted by the MCP. The MCP should have
standardized process for updating and maintaining this information.

19. Are Medicaid or CHIP fee schedules and contractual payment terms automated? Is payment against the
schedules automated for all types of participating providers?
Note (Q19): Manual payment processes are more prone to error and reduce processing speed.

END OF WORKSHEET A.1

APPENDIX A | A.35

Worksheet A.2. Information System Review Worksheet & Interview Guide
Instructions. EQROs can use this managed care plan (MCP) Information System Review Worksheet & Interview
Guide (Worksheet A.2) to conduct interviews with MCP staff who completed the ISCA tool (Worksheet A.1), as well
as other MCP staff as needed. Worksheet A.2 is organized in an open-ended format by section to correspond to the
ICSA tool completed by the MCP. 92
Before the site visit with the MCP, EQRO staff should:
•

Review the ISCA Worksheet A.1 and attached documentation submitted by the MCP, including documentation
referenced in the Summary of Requested Documentation Checklist submitted with the ISCA tool.

•

Identify issues to address in follow-up interviews with MCP personnel and record the questions in this worksheet.
Revise prompts in Sections 1 through 5 as needed.

•

If the MCP’s information system has been formally assessed within the past 2 years, please review the copy of
the assessment report included with Worksheet A. Follow-up on only those sections of the assessment report that
are not covered or that may have changed since the formal assessment was conducted.

During the site visit, EQRO staff should:
•

Use the space in this Worksheet to record responses or document specific issues. It is not necessary to cover
every question in the ISCA Worksheet A.1 submitted by the MCP if responses are clear.

•

Revise this Worksheet, as needed, to provide additional space under each question to record issues and findings.

After the site visit, EQRO staff should:
•

Analyze findings from the ISCA Worksheet A.1 and this Worksheet and prepare a statement of findings about the
MCP’s information system.

Contact Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, E-mail address, and date of interview, if applicable.
MCP Name:
Contact Name:
Title:
Mailing address:
Phone number:
E-mail address:
Interview Date:

For the purposes of the Appendix A worksheets, the term MCP includes managed care organizations (MCOs),
prepaid inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs), but does not include
primary care case management (PCCM) entities, because 42 CFR 438.242, which is the basis for the requirement
that states ensure maintenance of health information systems, is only applicable to MCOs, PIHPs and PAHPs.

A.36 | APPENDIX A

Type of delivery system (check all that apply)

□ MCO □ PIHP □ PAHP □ LTSS
□ Other: specify
___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □
Medicaid and CHIP

APPENDIX A | A.37

Section 1. Background Information
List questions for discussion with the MCP
Potential prompts:
•

Managed Care Model

•

Year MCP was incorporated

•

Member enrollment

•

Formal information system capability assessment

•

Recent information system enhancements

MCP responses to follow-up questions

Additional information provided by the MCP

A.38 | APPENDIX A

Section 2. Information Systems: Data Processing Procedures & Personnel
List questions for discussion with the MCP
Potential prompts:
•

System or repository for Medicaid claims and encounter data

•

Programming language(s) to create Medicaid data extracts or analytic reports

•

Programmer training, time, experience, turnover

•

Standard software development methodology

•

Version control software

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | A.39

Section 3. Staffing
List questions for discussion with the MCP
Potential prompts:
•

Staffing productivity

•

Processor training

MCP responses to follow-up questions

Additional information provided by the MCP

A.40 | APPENDIX A

Section 4. Security
List questions for discussion with the MCP
Potential prompts:
•

Disaster recovery plan

•

Disaster recovery system

•

Testing, backup systems, and storage

•

Computer system security

•

Cloud-based security

•

System access

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | A.41

Section 5A. Data Acquisition Capabilities: Administrative Data
List questions for discussion with the MCP about administrative data (claims and encounters)
Potential prompts:
•

Data submission methods

•

Claims or encounter submissions

•

Claims or encounter types

•

Diagnoses and procedures

•

Principal and secondary diagnoses

•

Missing, incomplete, or invalid claim/encounter submission fields

•

Claim/encounter accuracy verification

•

Systems changes/updates

•

Medicaid and CHIP data retention

•

Medicaid and CHIP claim/encounter audit policy

•

Pended claims/encounters process and reconciliation

•

Claim handling and processes that precede adjudication

•

Performance monitoring standards for Medicaid claims/encounters and results

MCP responses to follow-up questions

Additional information provided by the MCP

A.42 | APPENDIX A

Section 5B. Data Acquisition Capabilities: Enrollment System
List questions for discussion with the MCP about the enrollment system
Potential prompts:
•

Changes/updates to the Medicaid enrollment data system

•

Changes/updates effect on data quality/completeness

•

Continuity of enrollee ID numbers after data system changes

•

Continuity of enrollee ID numbers if disenrolled and reenrolled

•

Linkage of claim/encounter data for Medicare/Medicaid dual eligibles

•

If using a traditional PCCM, PCCM data follows T-MSIS coding guidance

•

Timeliness of Medicaid and CHIP enrollment updates

•

Medicaid continuous enrollment

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | A.43

Section 5C. Data Acquisition Capabilities: Ancillary Systems
List questions for discussion with the MCP about ancillary systems
Potential prompts:
•

Use of vendor data to calculate Medicaid and CHIP quality measures

•

Subcontracted Medicaid or CHIP benefits adjudicated through a vendor's system

•

Quality and accuracy of vendor data

•

Use of vendor data in Medicaid or CHIP-related studies

•

Use of unilateral or bi-directional data linkages to health information exchanges, registries, state vital statistics,
public health data
○ How the MCP verifies the data and data exchange process for these data sources

MCP responses to follow-up questions

Additional information provided by the MCP

A.44 | APPENDIX A

Section 5D. Data Acquisition Capabilities: Integration and Control of Data for Performance
Measure Reporting
List questions for discussion with the MCP integration and control of data for performance measure
reporting
Potential prompts:
•

MCP integration of Medicaid and CHIP claims, encounter, membership, provider, vendor, and other data to
calculate performance rates

•

Consolidation of data sets for each Medicaid and CHIP measure collected

•

Process for consolidating claims/encounter, member, and provider data for Medicaid and CHIP

•

Performance measure reporting

•

Performance measure repository

•

Report generation

•

Bonuses and penalties

•

Bundled/global payments

•

Provider profiles/directories

•

Process for maintaining Medicaid fee schedules and provider compensation rules

MCP responses to follow-up questions

Additional information provided by the MCP

END OF WORKSHEET A.2

APPENDIX A | A.45

Appendix B. Sampling Approaches for EQR
Data Collection Activities
Background
Sampling is used frequently in EQR-related activity processes for validation
and analysis purposes, such as:
Validating performance improvement projects (PIPs) (Protocol 1)

•

Validating the performance measures included in managed care plans’
(MCPs’) quality assessment and performance improvement (QAPI)
programs (Protocol 2)

•

Validating the encounter data reported by the MCP (Protocol 5)

•

Administering or validating quality of care surveys (Protocol 6)

•

Calculating additional performance measures (Protocol 7)

•

Implementing additional PIPs (Protocol 8)

•

Conducting focus studies of health care quality (Protocol 9)

This appendix provides a brief overview of the types of sampling approaches
and guidance for determining minimum sample sizes for EQR data collection
activities. A statistician or other staff with expertise in sample design and
implementation should advise the EQRO on the appropriate sampling strategy
to be used for each activity.

Types of Sampling Approaches
Probability Sampling
Probability (or random) sampling methods leave selection of population units to
chance and not to convenience or preference on the part of the individuals
conducting the study or otherwise participating in the study. Probability
sampling removes systematic bias due to observed and unobserved differences
in the sampling units. There are several types of probability sampling methods:
•

APPENDIX B

•

Simple random sampling is a method where all members of the study
population are listed in the sampling frame (see Box B.1) and have an equal
chance of being selected for the sample from the sampling frame. One way
to select a simple random sample is to first assign all units in the sampling
frame a unique identifier. Next, random numbers are generated for each unit
using random number generators (available in statistical software or
1

products). The random numbers then dictate the order in which units from the sampling
frame appear. Units are selected for the sample taking the first n units in that random order,
where n is the desired sample size. Simple random sampling ensures that all members of the
target population have an equal chance of selection.
Box B.1. What is a Sampling Frame?
A sampling frame is the list from which the sample is drawn. It includes the universe of members of the target study
population, such as individuals, households, providers, or other population units that are eligible to be included in the
study. The completeness and accuracy of the sampling frame are key to the representativeness of the sample.

•

Systematic random sampling is a method where units are systematically selected starting
with a randomly selected first unit. Systematic sampling can be used when a sampling frame
is organized or ordered in a way that does not bias the sample. Bias can occur if, for example,
there is a cyclical or seasonal order to the data that happens to coincide with the sampling
interval, in which case the sample will not fully represent the sampling frame. To select a
systematic sample, first determine what the sampling interval (i) is by dividing the total units
in the sampling frame (N) by the number of units in the sample (n). For example, if there are
250 units in the sampling frame and the desired sample size is 25, then the sampling interval
(i) is 250/25 = 10. Use a random number generator to select a number (k) between 1 and i.
Then select the kth, (k + i)th, (k + 2i)th, etc. units from the frame until the end of the
sampling frame is reached and you have selected n units

•

Stratified random sampling controls the proportion of the sample from subgroups of the
target population called strata. This technique divides the population into specific strata or
subgroups where the units are, ideally, homogeneous (the same or similar) within a stratum
and heterogeneous (different) between strata with respect to certain characteristics (e.g., age,
ethnicity, or diagnosis). Stratified sampling requires weighting the sample when a
disproportionately larger number of units may be selected from one strata compared to others
(“oversampling”). Stratification is done both to improve the representativeness of the total
population’s characteristics and to provide information about the characteristics of interest
within subgroups. Stratification can be used to oversample certain subgroups or simply to
ensure that the sample ends up with the same proportion as the population with respect to
these subgroups. Once strata are identified and constructed, sampling must be conducted
within each strata, independently, using probability sampling

•

One-stage cluster sampling is used when a comprehensive sampling frame of all units is not
readily available or would be too much of a burden to construct, or when data collection
cannot occur across the entire population due to financial or operational constraints. Units in
the population are gathered or classified into groups called clusters (these groups are similar
to strata used for stratified random sampling). Unlike the stratified sampling method, the
groups ideally should be heterogeneous with respect to the measured characteristic (but
rarely are). And, unlike stratified sampling, once clusters are identified, a random sample of
clusters is selected for data collection, with data then collected from all units in the selected
clusters

B.2 | APPENDIX B

•

Two-stage cluster sampling is an adaptation of one-stage cluster sampling. As with onestage cluster sampling, a sample of clusters is selected. However, unlike one-stage cluster
sampling, within the clusters there is a second stage of sampling—units within the clusters
are randomly selected so that some but not all units are selected for data collection. Twostage cluster sampling is ideal for situations where you do not have or are unable to construct
a frame of all the units in the population and you also cannot collect data from all clusters nor
all units in the selected clusters due to financial, operational, or other constraints

Non-Probability Sampling
Non-probability sampling methods are used when subjects are scarce or hard to sample (no
sampling frame) and/or the study relies on volunteers. The sample is based on the choice of those
administering the survey rather than chance; therefore, some bias can be expected. The following
are types of non-probability sampling:
•

Convenience sampling includes sampled units that are readily available or convenient to
sample. An example of a convenience sample for a focus study on patient experience with
Medicaid providers could include all patients sitting in the waiting room in a primary care
office on any given day. As another example, a focus study on health behaviors could
involve approaching people at a shopping mall

•

Quota sampling includes sampled units with known characteristics in the same proportion as
in the population. For example, if a target population is 55 percent female and 45 percent
male, the quota sample requires a similar female/male distribution. Quota sampling is
considered a non-probabilistic version of a stratified sample, in which a population is
segmented into mutually exclusive subgroups and judgment is used to select a sample based
on a specified proportion

Though non-random sampling methods may be statistically analyzed, caution should be
exercised when making inferences to the study population because the sample was not drawn
randomly and therefore, may not be representative of the population. Considering the risk of
biased results and the challenges to statistical interpretation, non-probability sampling is
discouraged. However, at times, it can be an appropriate and efficient way of collecting needed
information.

Calculating Minimum Sample Sizes for EQR Data Collection
Activities
Many EQR-related activities may involve sampling for data collection, such as validating the
completeness and accuracy of encounter data, assessing the reliability and validity of
performance measures calculated using the hybrid method, and implementing a survey.
Statistical power is a function of the sample size, the statistical significance criterion, and the
magnitude of the effect in the population.

APPENDIX B | B.3

Table B.1 provides guidance for determining minimum sample sizes for EQR data collection
activities. The minimum sample sizes vary based on the magnitude of the proportion (or
percentage) of the effect of interest. EQROs may base the proportion on the current year’s
administrative rate or the prior year’s reported rate (column 1). When the rate is unknown,
researchers typically base the sample size on a proportion of 0.50. Researchers also typically use
a statistical significance criterion of p < 0.05, As shown in Table B.1, the minimum sample size
for a rate of 0.05 or 0.95 (5 percent and 95 percent) is 100, while the minimum sample size for a
rate of 0.50 (50 percent) is 411 (column 2). The 95 percent confidence interval (and
corresponding lower and upper bounds) indicates the range in which the true value is estimated
to lie.
Table B.1. Guidance for Minimum Sample Sizes for EQR Data Collection Activities

Proportion
0.05

Minimum sample size for EQR data
collection activities

95 percent confidence interval (lower
and upper bound)

100

0.043
(0.007-0.093)

0.10

159

0.047
(0.053-0.147)

0.15

219

0.047
(0.103-0.197)

0.20

270

0.048
(0.152-0.248)

0.25

313

0.048
(0.202-0.298)

0.30

348

0.048
(0.252-0.348)

0.35

380

0.048
(0.302-0.398)

0.40

398

0.048
(0.352-0.448)

0.45

409

0.048
(0.402-0.498)

0.50

411

0.048
(0.452-0.548)

0.55

409

0.048
(0.502-0.598)

0.60

398

0.048
(0.552-0.648)

0.65

380

0.048
(0.602-0.698)

B.4 | APPENDIX B

Proportion

Minimum sample size for EQR data
collection activities

95 percent confidence interval (lower
and upper bound)

348

0.048

0.70

(0.652-0.748)
0.75

313

0.048
(0.702-0.798)

0.80

270

0.048
(0.752-0.848)

0.85

219

0.047
(0.803-0.897)

0.90

159

0.047
(0.853-0.947)

0.95

100

0.043
(0.907-0.993)

Documenting Sampling Methods for EQR Data Collection
Activities
In general, the following information should be documented about sampling approaches used for
EQR data collection activities:
•

Definition of the population included in the data collection activity (such as denominator for
performance measure, target population for PIP or focus study, time frame for measurement)

•

Sampling approach (such as simple random sampling, systematic random sampling, stratified
random sampling, one-stage cluster sampling, two-stage cluster sampling, convenience
sampling, quota sampling)

•

Sampling frame (such as enrollment file, claims extract, patient roster)

•

Sample exclusions (if any)

•

Sample size (including method used to determine minimum sample size)

•

Potential biases and selection issues that may affect representativeness of the sample and
generalizability of the results
END OF APPENDIX B

APPENDIX B | B.5

Appendix C. Acronyms Used In the
Protocols
AAFP
AHRQ
CAHPS®
CHIP
CHIPRA

HPV
ICD-10
IS
ISCA
LOINC
LOS
LTSS
MACBIS
MCO
MCP

APPENDIX C

CMS
CPT®
DBMS
DOB
DR
DSM-IV
EHR
EQR
EQRO
EVV
FFP
FFS
HCBS
HCPCS
HEDIS®
HHS
HIE
HIPAA
HIT
HITECH

American Academy of Family Physicians
Agency for Healthcare Research and Quality
Consumer Assessment of Healthcare Providers and Systems
Children’s Health Insurance Program
Children’s Health Insurance Program Reauthorization Act of
2009
Centers for Medicare & Medicaid Services
Current Procedural Terminology
Database Management System
Date of Birth
Disaster Recovery
Diagnostic and Statistical Manual of Mental Disorders
Electronic Health Record
External Quality Review
External Quality Review Organization
Electronic Visit Verification
Federal Financial Participation
Fee-For-Service
Home and Community Based Services
Healthcare Common Procedure Coding System
Healthcare Effectiveness Data and Information Set
U.S. Department of Health & Human Services
Health Information Exchange
Health Insurance Portability and Accountability Act
Health Information Technology
Health Information Technology for Economic and Clinical
Health Act
Human Papillomavirus Vaccine
International Statistical Classification of Diseases and
Related Health Problems
Information System
Information Systems Capability Assessment
Logical Observation Identifier Names and Codes
Length of Stay
Long-Term Services and Supports
CMS Medicaid and CHIP Business Information System
Managed Care Organization
Managed Care Plan

MGMA
MHSIP
MLTSS
MRR
MSIS
NCPDP
NCQA
NQS
PAHP
PCCM-E
PCMH-A
PCP
PDSA
PHI
PIHP
PII
PIP
QA
QAPI
QI
QS
T-MSIS

Medical Group Management Association
Mental Health Statistics Improvement Program
Managed Long-Term Services and Supports
Medical Record Review
Medicaid Statistical Information System
National Council for Prescription Drug Programs
National Committee for Quality Assurance
National Quality Strategy
Prepaid Ambulatory Health Plan
Primary Care Case Management Entity
Patient Centered Medical Home Assessment
Primary Care Provider
Plan Do Study Act
Protected Health Information
Prepaid Inpatient Health Plan
Personally Identifiable Information
Performance Improvement Project
Quality Assurance
Quality Assessment and Performance Improvement
Quality Improvement
Quality Strategy
Transformed Medicaid Statistical Information System

END OF APPENDIX C

C.2 | APPENDIX C

Appendix D. External Quality Review
Glossary of Terms
Acceptable Error The maximum percentage of missing, surplus, or
Rate
erroneous records that the state accepts.
A specific set of instructions for carrying out a procedure
or solving a problem.

Bias

A systematic distortion in data collection, analysis, or
reporting of research findings.

Binary Variable

A discrete variable with only two categories.

Categorical
Variable

A non-numeric variable with a range of non-ordered,
qualitative values (or categories). The values may be
coded as numbers but should not be interpreted
numerically.

Children’s
Health Insurance
Program
Reauthorization
Act of 2009
(CHIPRA)

Reauthorized the Children’s Health Insurance Program
(CHIP) under Title XXI of the Social Security Act.
CHIPRA included provisions to strengthen the quality of
care provided to children and improve health outcomes of
children in Medicaid and CHIP. CHIPRA requires the
U.S. Department of Health & Human Services (HHS) to
identify and publish a core measure set of children’s
health care quality measures for voluntary use by state
Medicaid and CHIP programs (CMS Core Set of
Children’s Health Care Quality Measures for Medicaid
and the Children’s Health Insurance Program (CHIP) (the
Child Core Set) and the Core Set of Health Care Quality
Measures for Adults Enrolled in Medicaid (the Adult
Core Set). The Child Core Set includes a range of
children’s quality measures encompassing both physical
and mental health. The initial Child Core Set was released
in 2010, updated in 2013, and is updated annually
thereafter.

Claims Data

See “Encounter Data.”

Compliance
Review

A process to determine the extent to which Medicaid and
CHIP managed care plans (MCPs) are complying with the
Medicaid standards set forth at 42 CFR 438, subpart D
and 42 CFR 438.330, which are adopted by CHIP at 42
CFR 457.

APPENDIX D

Algorithm

Confidence
Level

The likelihood, expressed as a percentage, that percentage that a sample
finding is true for the population from which the sample was taken. For
example, a 95 percent confidence interval indicates a 5 percent chance that
the sample result is due to chance and is not true for the population.

Consumer
Assessment of
Healthcare
Providers and
Systems
(CAHPS®)

A series of consumer and patient surveys rating health care experiences in
the U.S. All surveys officially designated as CAHPS® surveys have been
approved by the CAHPS® Consortium, which is overseen by the Agency for
Healthcare Research and Quality (AHRQ). CAHPS® surveys are an integral
part of CMS’ efforts to improve health care in the U.S. CAHPS® surveys
follow scientific principles in survey design and development, are designed
to reliably assess the experiences of a large sample of patients, and use
standardized questions and data collection protocols to ensure that
information can be compared across health care settings.

Continuous
Variable

A numeric variable with a range of numerical values. Data collected for a
continuous variable may be recoded as a discrete variable.

Correlation
Coefficient

A statistical measure of the interdependence of two random variables, the
value of which indicates how much a change in one variable is related to a
change in the other variable. Correlation coefficients range in value from -1
to +1. A perfect positive correlation is +1 and a perfect negative correlation
is -1. Zero indicates the absence of a relationship between the variables.

CPT®

A coding system, defined in the American Medical Association publication
“Current Procedural Terminology”, for medical procedures that are used for
billing and quality measures.

Database
Management
System
(DBMS)

System software for creating, managing, and maintaining databases.

Definition of
Network
Adequacy
Indicator

A clear description of the network adequacy indicator, including criteria for
calculating the numerator and denominator. The definition should address
specific methodological issues that impact indicator calculations. For
example, for time and distance indicators, the definition should specify
whether distance is measured “as the crow flies” or using driving distances.
The definition should also identify the provider types to which the indicator
applies.

D.2 | APPENDIX D

Denominator

The bottom part of the fraction that represents the total number of parts
created from the whole. For the purposes of the EQR protocols, the
denominator provides the general specifications of any clinical component
that is the basis for inclusions and exclusions in the population to be
considered in a measure.

Discrete
Variable

A numeric variable with a limited number of possible categories. A binary
variable is a type of discrete variable with only two categories.

Edit Checks

A program instruction that tests the quality and validity of data entered.

Encounter Data

The managed care equivalent of fee-for-service (FFS) claims. Encounter
data is the information related to the receipt of any item or service by a
beneficiary enrolled in a managed care plan (MCP). They reflect that a
provider rendered a specified service under a managed care delivery system,
regardless of if or how the MCP ultimately reimbursed the provider.
Encounter data include substantially the same information included on claim
forms (e.g., UB-04 or CMS 1500), although not necessarily in the same
format. Providers submit claims or encounters to MCPs for service(s)
rendered that would traditionally be submitted as claims in a FFS system.

Enrollee

An eligible individual who is covered by a managed healthcare plan. A
beneficiary is an eligible individual who receives health care insurance
through the Medicare or Medicaid programs.

EQR-Related
Activities

The activities addressed in these protocols. EQR-related activities may be
conducted by the state, its agent that is not an MCO, PIHP, PAHP, or PCCM
entity (described at 42 CFR 438.310(c)(2)), or an EQRO. See 42 CFR
438.358.

Erroneous
Encounters

Encounters that occurred and are represented by an encounter record that
contains incorrect data elements.

External Quality The analysis and evaluation by an external quality review organization
Review (EQR) (EQRO), of aggregated information on quality, timeliness, and access to the
health services that an MCO, PIHP, PAHP, or PCCM entity (described at 42
CFR 438.310(c)(2)), or their contractors furnish to Medicaid beneficiaries.
External Quality An organization that meets the competence and independence requirements
Review
set forth at 42 CFR 438.354, and performs external quality review or other
Organization
EQR-related activities as set forth in 42 CFR 438.358, or both.
(EQRO)
Fee-for-Service

A payment mechanism in which payment is made for each service used.

Focus Study

A study of a particular aspect of clinical care or nonclinical services
provided by a managed care plan (MCP) at a point in time. See 42 CFR
438.358(c)(5).
APPENDIX D | D.3

Generalizability The extension of findings and conclusions from a study sample to the
population from which the sample was drawn.
Healthcare
Common
Procedural
Terminology
(HCPCS)

A standardized coding system for describing the specific items and services
provided in the delivery of health care. HCPCS contains levels of codes,
including the American Medical Association’s CPT® and alphanumeric
codes for non-physician services, items, and supplies not contained in
CPT®.

Healthcare
Effectiveness
Data and
Information Set
(HEDIS®)

A collection of standardized performance measures and their definitions
designed to ensure that purchasers and consumers can reliably compare the
performance of managed health care plans. The performance measures are
related to public health issues such as cancer, heart disease, and asthma and
also include well-child visits. HEDIS® is sponsored, supported, and
maintained by the National Committee for Quality Assurance (NCQA).

Health
Information
Technology
(HIT)

Used by health care providers to manage patient care and health through
using and sharing health information in a secure system. EHR, meaningful
use, and mobile health laws and regulations all fall under the umbrellas of
HIT.

Hybrid Data

Administrative data supplemented by medical record review.

Improvement
Strategy

An intervention designed to change behavior at the member, provider, and/or
managed care plan (MCP)/system level.

Indicator

An observable and measurable characteristic that can be used to show
changes or progress over time toward achieving a specific outcome.

Information
System
Capabilities
Assessment
(ISCA)

Assessment of the desired capabilities of the MCP’s information system
which poses standard questions used to assess the strength of the system; this
provides information to the EQRO about the extent to which the information
system is capable of producing valid encounter data, performance measures,
and other data necessary to support quality assessment and improvement, as
well as managing the care delivered to its beneficiaries. Please refer to
Appendix A. Information System Capabilities Assessment for more
information.

Kappa statistic

A test statistic that measures interrater reliability for categorical data (e.g.,
sex, gender, race, etc.).

Locating

Locating is a technique used to improve response rates by locating and
contacting sample members. This includes verified collection of data, such
as first and last name, home address, email address, phone number(s), date
of birth, language preference, etc.

D.4 | APPENDIX D

Managed Care
Plans (MCPs)

For the purposes of the EQR protocols, encompasses managed care
organizations (MCOs), prepaid inpatient health plans (PIHPs), prepaid
ambulatory health plans (PAHPs), and the subset of primary care case
management (PCCM) entities described in 42 CFR 438.310(c)(2).

Managed Care
See “State Quality Strategy.”
Quality Strategy
Margin of Error A statistic expressing the amount of random sampling error in a survey's
results. The larger the margin of error, the less faith one should have that the
sample result is the true population value.
Measure

A standard used for valuing or determining the extent or quantity of
something.

Missing
Encounters

Encounters that occurred but are not represented by an encounter record.

Network
Adequacy
Indicators

Metrics used to measure adherence to network adequacy standards and to
determine plan compliance with state network adequacy standards. For the
example given above, the network adequacy indicator may be the proportion
of enrollees who have access to a PCP within 30 miles or 30 minutes of their
home.

Network
Adequacy
Standards

Quantitative parameters that states establish to set expectations for
contracted managed care plans’ provider networks. For example, a state may
set a network adequacy standard that all enrollees have access to a primary
care provider (PCP) within 30 miles or 30 minutes of their home.

Non-Probability Methods that are used when subjects are scarce or hard to sample (no
Sampling
sampling frame) and/or the study relies on volunteers. The sample is based
on the choice of those administering the survey rather than chance; therefore,
some bias can be expected. Non-probability sampling includes convenience
sampling and quota sampling. Please refer to Appendix B. Sampling
Approaches for EQR Data Collection Activities for more information.
Numerator

The top part of the fraction that represents how many parts of that whole are
being considered. For example, with large population of patients, the
numerator would be the number of patients in a study meeting the
specifications of a clinical component in a measure.

Pay for
Performance

An umbrella term for initiatives aimed at improving the quality, efficiency,
and overall value of health care. These arrangements provide financial
incentives to hospitals, physicians, and other health care providers for
improvements in quality of care and health outcomes for patients.

APPENDIX D | D.5

Pearson
Correlation
Coefficient
(also,
Pearson’s)

The most common measure of correlation in statistics. It shows the linear
relationship between two variables X and Y, with results between -1 and +1,
where 1 is total positive linear correlation, 0 is no linear correlation, and -1 is
negative linear correlation. The closer the value to zero, the greater the
variation the data points are around the line of best fit.

Performance
Improvement
Project (PIP)

A project that implements an intervention designed to achieve and sustain
significant improvement in health outcomes over time.

Performance
Measure

Used to monitor performance at a point in time, track performance over
time, compare performance, and inform decisions. For the purposes of the
EQR protocols, it refers to monitoring the performance of individual
managed care plans (MCPs) at a point in time, to track MCP performance
over time, to compare performance among MCPs, and to inform the
selection and evaluation of quality improvement activities.

Plan Do Study
Act (PDSA)

A continuous cycle of measuring and analyzing performance. For the
purposes of the EQR protocols, PDSA cycles refer to testing changes on a
small scale and applying rapid-cycle learning principles to adjust
intervention strategies over the course of time (such as in PIPs).

Prepaid
Ambulatory
Health Plan
(PAHP)

An entity that provides services to enrollees under contract with the state and
on the basis of capitation payments or other payment arrangement’s that do
not use state plan payment rates; does not provide or arrange for and is not
otherwise responsible for the provision of any inpatient hospital or
institutional services for its enrollees; and does not have a comprehensive
risk contract.

Prepaid
A prepaid health plan that provides services to enrollees under contract with
Inpatient Health the state and on the basis of capitation payments or other payment
Plan (PIHP)
arrangements that do not use State plan payment rates; provides, arranges
for, or otherwise has responsibility for the provision of any inpatient hospital
or institutional services for its enrollees; and does not have a comprehensive
risk contract.
Primary Care
Case
Management
(PCCM)

A system under which a primary care case manager contracts with the state
to furnish case management services (which include the location,
coordination and monitoring of primary health care services) to Medicaid
beneficiaries.

Primary Care
Case
Management
(PCCM) Entity

The term PCCM entity in these EQR protocols only applies to those PCCM
entities whose contracts with a state provide for shared savings, incentive
payments, or other financial reward for the PCCM entity for improved
quality outcomes, as described at 42 CFR 438.310(c)(2).

D.6 | APPENDIX D

Probability (or
random)
Sampling

Refers to sampling methods that leave selection of population units to
chance and not to convenience or preference on the part of the individuals
conducting the study or otherwise participating in the study. Probability
sampling removes systematic bias in the selected sample due to observed
and unobserved differences in the sampling units. Types of probability
sampling include simple random sampling, systematic random sampling,
stratified random sampling, one-stage cluster sampling, and two-stage
cluster sampling. Please refer to Appendix B. Sampling Approaches for
EQR Data Collection Activities for more information.

Programmatic
Significance

The practical effect or importance of an intervention implemented through a
program or specified method.

Protected
Health
Information

A class of patient data that can be linked to a specific individual.

Quality

The degree to which an MCO, PIHP, PAHP, or PCCM entity (described at
42 CFR 438.310(c)(2)) increases the likelihood of desired health outcomes
of its enrollees through structural and operational characteristics, the
provision of services that are consistent with current professional, evidencebased knowledge, and interventions for performance improvement.

Quality
Assurance Plan

A plan that includes processes to monitor, evaluate and review all aspects of
the survey administration procedures. The purpose of a quality assurance
plan is to document reviews and audits to ensure appropriate processes are
correctly followed.

Registry Data

Clinical data that is recorded about the health status of patients and health
care they receive over time. This data is maintained in a clinical data
registry.

Reliability

Refers to (1) the internal consistency of a study instrument, and (2) that data
are producing consistent results.

Sample

A subset selected from a population.

Sampling
Frame

The list from which the sample is drawn. It includes the universe of
members of the target study population, such as individuals, households,
encounters, providers, or other population units that are eligible to be
included in the study. The completeness, recency, and accuracy of the
sampling frame are key to the representativeness of the sample.

Significant
Improvement

A measurable, statistically significant change in performance related to an
intervention.

APPENDIX D | D.7

State Quality
Strategy

A strategy to assess and improve the quality of Medicaid managed care
services within a state, per 42 CFR 438.340 and adopted by CHIP at 42 CFR
457.1240(e).

Statistical
Significance

A measure of whether research findings are meaningful. More specifically,
whether results match closely to what one would expect to find in an entire
population. The test for statistical significance requires (1) deciding an alpha
level, meaning, the error rate (typically 5 percent or less), (2) collecting data,
(3) calculating the test statistic, and (4) comparing the calculated test statistic
with a statistic from a statistical table.

Study
Population

The population identified for the study. It may include the entire population
or a sample of the population depending on the nature of the study question
and available data.

Study Question

Identifies the focus of the study and sets the framework for data collection
and analysis. The study question should be clear, concise, and answerable.

Study Variable

A measurable characteristic, quality, trait, or attribute of a particular
individual, object, or situation being studied.

Sustained
Improvement

Significant changes in processes or performance as demonstrated through
repeated measurements over comparable time periods using the same
methodology as in the baseline measurement.

Target
Population

The group of individuals that are the intended recipient of a particular
service or intervention.

Transformed
Medicaid
Statistical
Information
System (TMSIS)

A critical data and systems component of the CMS Medicaid and CHIP
Business Information System (MACBIS). CMS has been working with
states to transform the MSIS system, which was used to (1) collect
utilization and claims data as well as other key Medicaid and CHIP program
information, (2) keep pace with the data needed to improve beneficiary
quality of care, (3) assess beneficiary care and enrollment, (4) improve
program integrity, and (5) support states, the private market, and
stakeholders with key information. The T-MSIS data set contains (1)
enhanced information about beneficiary eligibility, (2) beneficiary and
provider enrollment, (3) service utilization, (4) claims and managed care
data, and (5) expenditure data for Medicaid and CHIP.

T-test

Most commonly used with small sample sizes, this test asks whether a
difference between two samples/groups’ averages is unlikely to have
occurred because of random chance in sample selection. A difference is
more likely to be meaningful or if (1) the difference between the averages is
large, (2) the sample size is large, and (3) the standard deviation is low.

Unit of Analysis The entity (“what” or “whom”) that is being studied.

D.8 | APPENDIX D

Validation

The review of information, data, and procedures to determine the extent to
which it is accurate, reliable, free from bias, and meets standards for data
collection.

Validity

The degree to which a tool measures what it is intended to measure.

Verification

The internal review of documentation, data, measures, and assessments to
determine if measurements are accurate.

Vital Records

Records of life events kept under government authority. These include life
events such as birth certificates, marriage licenses, and death certificates.

END OF APPENDIX D

APPENDIX D | D.9

File Type	application/pdf
File Title	CMS External Quality Review (EQR) Protocols
Subject	External Quality Review, EQR; EQRO; Protocols; Medicaid; Managed Care; CHIP; CMS; Quality: Medicaid managed care rule:, Network
Author	Centers for Medicare & Medicaid Services
File Modified	2023-02-16
File Created	2023-01-27