Form CMS-R-305 External Quality Review (EQR) Protocols

External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364 (CMS-R-305)

R4_CMS_2018_External_Quality_Review_Protocols_07262019

(State Gov't) External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB: 0938-0786

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0938-0786 can be found here:

Document [pdf]

Download: pdf | pdf

OMB Control No. 0938-0786
Expires: TBD

CMS EXTERNAL
QUALITY REVIEW (EQR)

PROTOCOLS
July 2019

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB
control number. The valid OMB control number for this information collection is 0938-0786. The time required to complete this information collection is
estimated to average 7,688 hours per response for state governments and 185 hours per response for the private sector, including the time to review
instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports
Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

This page left blank for double-sided copying.

CONTENTS
INTRODUCTION........................................................................................................................................... 7
Background ...................................................................................................................................... 7
Overview of the EQR Protocols ..................................................................................................... 13
Considerations Before Conducting EQR-Related Activities .......................................................... 15
Nonduplication for Mandatory EQR-Related Actvities ................................................................... 16
Tips for Drafting EQR Technical Reports....................................................................................... 18
Getting Started on the EQR Protocols ........................................................................................... 21
For Further Information .................................................................................................................. 25
PROTOCOL 1. VALIDATION OF PERFORMANCE IMPROVEMENT PROJECTS ................................... 27
Background .................................................................................................................................... 27
Getting Started on Protocol 1 ......................................................................................................... 28
Activity 1: Assess the PIP Methodology ........................................................................................ 29
Activity 2: Perform Overall Validation and Reporting of PIP Results ............................................. 37
Activity 3: Verify PIP Findings (Optional) ....................................................................................... 38
Worksheets for Protocol 1: PIP Validation Tools and Reporting Framework ................................ 40
PROTOCOL 2. VALIDATION OF PERFORMANCE MEASURES ............................................................. 57
Background .................................................................................................................................... 57
Getting Started on Protocol 2 ......................................................................................................... 58
Activity 1: Conduct Pre-Onsite Visit Activities ................................................................................ 59
Activity 2: Conduct Onsite Visit Activities ....................................................................................... 68
Activity 3: Conduct Post-Onsite Visit Activities .............................................................................. 72
Worksheets for Protocol 2: Performance Measure Validation Tools ............................................. 75
PROTOCOL 3. REVIEW OF COMPLIANCE WITH MEDICAID AND CHIP MANAGED CARE
REGULATIONS ........................................................................................................................... 115
Background .................................................................................................................................. 115
Getting Started on Protocol 3 ....................................................................................................... 117
Activity 1: Establish Compliance Thresholds ............................................................................... 119
Activity 2: Perform the Preliminary Review (Pre-Onsite Visit) ..................................................... 119
Activity 3: Conduct MCP Onsite Visit ........................................................................................... 121
Activity 4: Compile and Analyze Findings (Post-Onsite Visit) ...................................................... 125

CONTENTS | iii

Activity 5: Report Results to the State ......................................................................................... 126
Worksheets for Protocol 3: Compliance Review Tools ................................................................ 128
PROTOCOL 4. VALIDATION OF NETWORK ADEQUACY..................................................................... 197
Reserved. ..................................................................................................................................... 197
PROTOCOL 5. VALIDATION OF ENCOUNTER DATA REPORTED BY THE MEDICAID AND
CHIP MANAGED CARE PLAN .................................................................................................... 199
Background .................................................................................................................................. 199
Getting Started on Protocol 5 ....................................................................................................... 200
Activity 1: Review State Requirements ........................................................................................ 202
Activity 2: Review the MCP’s Capability ...................................................................................... 203
Activity 3: Analyze Electronic Encounter Data ............................................................................. 205
Activity 4: Review Medical Records ............................................................................................. 210
Activity 5: Submit Findings ........................................................................................................... 213
Worksheets for Protocol 5: Encounter Data Tables ..................................................................... 214
PROTOCOL 6. ADMINISTRATION OR VALIDATION OF QUALITY OF CARE SURVEYS .................... 225
Background .................................................................................................................................. 225
Getting Started on Protocol 6 ....................................................................................................... 226
Section I. Administering a Survey ................................................................................................ 228
Activity I.1: Identify the Survey Purpose, Objectives, and Audience ........................................... 228
Activity I.2: Develop a Work Plan ................................................................................................. 229
Activity I.3: Select the Survey Instrument .................................................................................... 230
Activity I.4: Develop the Sampling Plan ....................................................................................... 233
Activity I.5: Develop a Strategy to Maximize Response .............................................................. 235
Activity I.6: Develop a Quality Assurance Plan ............................................................................ 237
Activity I.7: Implement the Survey According to the Work Plan ...................................................... 238
Activity I.8: Prepare and Analyze Survey Data and Present Results in a Final Report ............... 239
Section II. Validating a Survey ..................................................................................................... 241
Activity II.1: Review the Survey Purpose, Objectives, and Audience .......................................... 242
Activity II.2: Review the Work Plan .............................................................................................. 242
Activity II.3: Review the Reliability and Validity of the Survey Instrument ................................... 242
Activity II.4: Review the Sampling Plan........................................................................................ 243
Activity II.5: Review the Adequacy of the Response Rate ........................................................... 244
Activity II.6: Review the Quality Assurance Plan ......................................................................... 244

iv | CONTENTS

Activity II.7: Review the Survey Implementation .......................................................................... 245
Activity II.8: Review the Survey Data Analysis and Final Report ................................................. 245
Technical Appendix for Protocol 6: Understanding Potential Sources of Survey Error ............... 246
Worksheets for Protocol 6: Survey Administration and Validation Tools ..................................... 248
PROTOCOL 7. CALCULATION OF ADDITIONAL PERFORMANCE MEASURES ................................ 257
Background .................................................................................................................................. 257
Getting Started on Protocol 7 ....................................................................................................... 258
Activity 1: Prepare for Measurement............................................................................................ 259
Activity 2: Calculate Measures ..................................................................................................... 261
Activity 3: Report Results ............................................................................................................. 264
Worksheets for Protocol 7: Performance Measure Calculation Tools ......................................... 266
PROTOCOL 8. IMPLEMENTATION OF ADDITIONAL PERFORMANCE IMPROVEMENT
PROJECTS .................................................................................................................................. 273
Background .................................................................................................................................. 273
Getting Started on Protocol 8 ....................................................................................................... 274
Activity 1: Select the PIP Topic .................................................................................................... 275
Activity 2: Define the PIP Aim Statement..................................................................................... 276
Activity 3: Identify the PIP Population .......................................................................................... 276
Activity 4: Use Sound Sampling Methods .................................................................................... 276
Activity 5: Select the PIP Variables ............................................................................................. 277
Activity 6: Collect Valid and Reliable Data ................................................................................... 278
Activity 7: Analyze Data and Interpret Results ............................................................................. 278
Activity 8: Review Improvement Strategies.................................................................................. 278
Activity 9: Assess Whether Significant and Sustained Improvement Occurred........................... 279
PROTOCOL 9. CONDUCTING FOCUS STUDIES OF HEALTH CARE QUALITY ................................. 281
Background .................................................................................................................................. 281
Getting Started on Protocol 9 ....................................................................................................... 282
Activity 1: Select the Study Topic(s) ............................................................................................ 283
Activity 2: Define the Study Question(s) ...................................................................................... 284
Activity 3: Select the Study Variable(s) ........................................................................................ 284
Activity 4: Develop a Plan to Study the Population ...................................................................... 286
Activity 5: Collect Data ................................................................................................................. 287
Activity 6: Analyze and Interpret Study Results ........................................................................... 288

CONTENTS | v

Activity 7: Report Results to the State ......................................................................................... 289
PROTOCOL 10. ASSIST WITH QUALITY RATING OF MEDICAID AND CHIP MANAGED CARE
ORGANIZATIONS, PREPAID INPATIENT HEALTH PLANS, AND PREPAID
AMBULATORY HEALTH PLANS ................................................................................................. 291
Reserved. ..................................................................................................................................... 291
APPENDIX A. INFORMATION SYSTEMS CAPABILITIES ASSESSMENT ............................................ 293
Background .................................................................................................................................. 293
Getting Started on the ISCA ........................................................................................................ 296
Activity 1: MCP Completes the ISCA Tool ................................................................................... 297
Activity 2: Perform Preliminary ISCA Review (Pre-Onsite Analysis) ............................................... 297
Activity 3: Conduct MCP Onsite Visit ........................................................................................... 297
Activity 4: Compile and Analyze ISCA Findings .......................................................................... 298
Activity 5: Draft ISCA Summary for EQR Technical Report......................................................... 298
The Future of Information Systems Assessment ......................................................................... 298
Worksheets for Appendix A ......................................................................................................... 301
APPENDIX B. SAMPLING APPROACHES FOR EQR DATA COLLECTION ACTIVITIES ..................... 337
Background .................................................................................................................................. 337
Types of Sampling Approaches ................................................................................................... 337
Calculating Minimum Sample Sizes for EQR Data Collection Activities ...................................... 339
Documenting Sampling Methods for EQR Data Collection Activities .......................................... 341
APPENDIX C. ACRONYMS USED IN THE PROTOCOLS ...................................................................... 343
APPENDIX D. EXTERNAL QUALITY REVIEW GLOSSARY OF TERMS ................................................ 345

vi | CONTENTS

INTRODUCTION
BACKGROUND

Key Definitions
• Managed care plan (MCP). Encompasses managed care organizations (MCOs),
prepaid inpatient health plans (PIHPs), prepaid ambulatory health plans (PAHPs), and
primary care case management (PCCM) entities described in 42 C.F.R. § 438.310(c)(2).
• External quality review (EQR). EQR is the analysis and evaluation of aggregated
information on quality, timeliness, and access to the health services that an MCP or its
contractors furnish to Medicaid beneficiaries [see 42 C.F.R. § 438.320]. EQR can only
be conducted by a qualified EQRO.
• External quality review organization (EQRO). An EQRO is an organization that
meets the competence and independence requirements set forth in 42 C.F.R. §
438.354, and performs EQR, EQR-related activities, or both.
• EQR-related activities. The activities addressed in these protocols. EQR-related
activities produce the data used by an EQRO to complete the annual EQR. EQRrelated activities may be conducted by the state, its agent that is not an MCP, or an
EQRO [see 42 C.F.R. § 438.358].

The federal requirements related to Medicaid managed care quality were
established in statute at section 1932(c) of the Social Security Act (the
Act) and are set forth in 42 C.F.R. § 438, subpart E. The same statutory
federal requirements were made applicable to CHIP managed care
quality through section 2103(f)(3) of the Act and are set forth in 42 C.F.R.
§§ 457.1240 and 1250. The timeline in Figure 1 chronicles the evolution
of the scope of EQR in Medicaid and CHIP.

Estimates are for December 2016. December 2016 Medicaid and CHIP Enrollment Data
Highlights available at https://www.medicaid.gov/medicaid/program-information/medicaid-andchip-enrollment-data/report-highlights/index.html.
1

INTRODUCTION

Together, Medicaid and the Children’s Health Insurance Program (CHIP)
cover approximately 74 million people, 1 including nearly one of every two
children 2 and nearly half of all births. 3 About 70 percent of adults and
children in Medicaid and CHIP obtain their care through managed care
plans (MCPs), although the rate of managed care enrollment in states using
a managed care delivery system varies widely.4 (See box, Key Definitions.)

Medicaid/CHIP enrollment data for FFY 2015 are available at
https://www.medicaid.gov/chip/downloads/fy-2015-childrens-enrollment-report.pdf. Enrollment
data reflect the number of children age 21 and younger ever enrolled during the year. Agespecific population data for children age 21 and younger as of July 2015 are available at:
https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=PEP_2015_P
EPSYASEX&prodType=table.
3
Markus, A.R., E. Andres, K. West, N. Garro, and C. Pellegrini. “Medicaid Covered Births, 2008
Through 2010, in the Context of Implementation of Health Reform.” Women’s Health Issues,
vol. 23, issue 5, 2013, pp. e273–e280. Available at http://www.whijournal.com/article/S10493867%2813%2900055-8/abstract.
4
More information about Medicaid managed care is available at
https://www.medicaid.gov/medicaid/quality-of-care/medicaid-managed-care/index.html.
2

Figure 1. Evolution of EQR in Medicaid and CHIP

Notes: CHIP = Children’s Health Insurance Program; CMS = Centers for Medicare & Medicaid Services; EQR = external quality
review; HHS = U.S. Department of Health & Human Services; MCOs = managed care organizations; PAHPs = prepaid ambulatory
health plans; PCCM = primary care case management; PIHPs = prepaid inpatient health plans.
a

Balanced Budget Act of 1997 amending section 1932(c)(1)(B) of the Social Security Act.

Section 1139A(c)(2) of the Social Security Act, as amended by section 401(a) of CHIPRA, requires the HHS Secretary to
summarize State-specific information on the quality of health care furnished to children under titles XIX (Medicaid) and XXI (CHIP).
Section 1139A(c)(1)(B) of the Act specifically requests information gathered from the external quality reviews of managed care
organizations (MCOs) and benchmark plans.

The Centers for Medicare & Medicaid Services (CMS) published the Medicaid and CHIP
managed care final rule in May 2016, which aligns key rules with those of other health insurance
coverage programs, modernizes how states purchase managed care for beneficiaries, and
strengthens the consumer experience and key consumer protections. 5 The rule also updated
and expanded EQR in the following ways:
•

Clarified that the Children’s Health Insurance Program Reauthorization Act of 2009
(CHIPRA) applied EQR (including EQR-related activities) to both separate CHIP MCPs and
Medicaid Expansion CHIP MCPs. A state that uses MCPs to provide CHIP benefits must
develop and implement a managed care quality strategy and must require CHIP MCPs to
operate quality assessment and performance improvement (QAPI) programs

More information about the Medicaid and CHIP managed care final rule is available at
https://www.medicaid.gov/medicaid/managed-care/guidance/final-rule/index.html and the Federal Register at
https://www.gpo.gov/fdsys/pkg/FR-2016-05-06/pdf/2016-09581.pdf.

8 | INTRODUCTION

•

Applied EQR to a broader range of Medicaid MCPs, that is, beyond managed care
organizations (MCOs) and prepaid inpatient health plans (PIHPs) to also include prepaid
ambulatory health plans (PAHPs) and primary care case management (PCCM) entities
whose contracts with the state provide for shared savings, incentive payments or other
financial reward for the PCCM entity for improved quality outcomes 6

•

Added two EQR-related activities: (1) a mandatory EQR-related activity, validation of
network adequacy (effective no later than one year from the issuance of the associated
EQR protocol) and (2) an optional EQR-related activity, assistance with the quality rating of
MCOs, PIHPs, and PAHPs required under a Medicaid and CHIP quality rating system
(effective no earlier than the issuance of the associated EQR protocol) 7

EQR is one part of an interrelated set of quality requirements that apply to Medicaid managed
care. For example, per 42 C.F.R. §§ 438.364(a)(4) and 457.1250, the feedback obtained from
the state’s EQRO should be used by states when they examine and update their quality strategy
(Figure 2). 8 States’ quality strategies, in turn, are implemented through the ongoing
comprehensive quality assessment and performance improvement (QAPI) program 9 that
contracted MCPs are required to establish for the services the MCP furnishes to its enrollees.
The performance improvement projects (PIPs) and performance measures included in QAPIs
are, in turn, validated through the annual EQR. Therefore, it is important that states ensure
alignment among the MCPs’ QAPI requirements, the state’s quality strategy, and the annual
EQR activities.
Figure 2. Relationship between the external quality review, state quality strategy, and
QAPI program

Notes: EQR = external quality review; QAPI = quality assessment and performance improvement; QS = quality strategy.

For the purposes of these protocols, all references to PCCM entities should be assumed to refer to the applicable subset of
PCCM entities described at 42 C.F.R. §§ 438.310(c)(2), and 457.1240(f).

Page 27499 of the final rule includes effective dates for the additional EQR-related activities: “States must begin
conducting the EQR-related activity described in § 438.358(b)(1)(iv) (relating to the mandatory-EQR-related activity of the
validation of network adequacy) no later than one year from the issuance of the associated EQR-protocol. States may begin
conducting the EQR-related activity described in § 438.358(c)(6) (relating to the optional EQR-related activity of plan rating)
no later than the issuance of the associated EQR protocol.”

8
9

See 42 C.F.R. § 438.340 (as cross referenced at §457.1250 for CHIP).
See 42 C.F.R. § 438.330 (as cross referenced at §457.1250 for CHIP).

INTRODUCTION | 9

States using a managed care delivery system for all or some of their Medicaid and/or CHIP
beneficiaries are required to contract with a qualified independent external quality review
organization (EQRO) to conduct an annual external quality review (EQR) to assess and monitor
the quality of care provided to Medicaid and CHIP beneficiaries enrolled in MCPs and to identify
opportunities for quality improvement. 10 To simplify the narrative of these protocols, the term
“EQRO” is used to refer to the entity which conducts the EQR-related activities that generate the
information for the annual EQR. An EQRO is the only entity which may conduct the annual
EQR, that is, the analysis and evaluation of information generated by the EQR-related activities
(or via nonduplication, if applicable) regarding the quality, timeliness, and access to the health
care services that an MCP, or its contractors, furnish to beneficiaries. The end product of the
EQR is an annual EQR technical report, which summarizes findings on access and quality of
care, and must be drafted by said EQRO. 11
Figure 3. The EQR Process

Notes: CHIP = Children’s Health Insurance Program; CMS = Centers for Medicare & Medicaid Services; EQR = external quality
review; EQRO = external quality review organization; MCOs = managed care organizations; PAHPs = prepaid ambulatory health
plans; PCCM = primary care case management; PIHPs = prepaid inpatient health plans.

States with both Medicaid and CHIP managed care programs may elect to contract with a single
EQRO to conduct EQR of both Medicaid and CHIP or may contract with different EQROs for
EQR of Medicaid and CHIP. Many states choose to utilize the same EQRO for EQR of both
Medicaid and CHIP.

For more information on state contract options for EQR, see 42 C.F.R. § 438.356 (as cross referenced at § 457.1250 for
CHIP).

For more information on the EQR technical report, see 42 C.F.R. § 438.364 (as cross referenced at § 457.1250 for CHIP).

10 | INTRODUCTION

The EQR process includes a series of mandatory and
optional EQR-related activities designed to provide a
sound understanding of the strengths and weaknesses
of Medicaid and CHIP MCP performance related to
quality, timeliness, and access to care (See box,
Mandatory and Optional EQR-Related Activities). The
EQR-related activities are intended to (1) improve
states’ ability to oversee and manage the MCPs they
contract with for services, and (2) help MCPs improve
their performance with respect to quality, timeliness,
and access to care. Effective implementation of the
EQR-related activities will facilitate state efforts to
purchase high-value care (rather than volume) and to
achieve higher performing health care delivery systems
for their Medicaid and CHIP beneficiaries. States have
flexibility regarding who will conduct the EQR-related
activities; they may be conducted by the state, its agent
that is not a managed care plan, or an EQRO. If the
state elects to contract with an EQRO to conduct the
EQR-related activities, this can be the same EQRO
that conducts the EQR for the state, or one or more
additional EQROs. 12

Mandatory EQR-Related Activities
• Validate performance improvement
projects (PIPs)
• Validate performance measures
• Review compliance with Medicaid and
CHIP managed care regulations
• Validate network adequacy (Reserved)
Optional EQR-Related Activities
• Validate encounter data reported by
MCPs
• Administer or validate quality of care
surveys
• Calculate additional performance
measures
• Conduct additional PIPs
• Conduct focus studies of health care
quality
• Assist with quality rating of MCPs
(Reserved)

Medicaid and CHIP MCOs, PIHPs, and PAHPs are subject to all four mandatory EQR-related
activities; 13 PCCM entities are subject to two of the mandatory EQR-related activities (a
compliance review and validation of performance measures). 14 See Table 1 for additional
information regarding the application of EQR-related activities to MCPs.
Table 1. Application of Mandatory and Optional EQR-related activities by MCP type
MCP Type
EQR-Related Activity

MCO

PIHP

PAHP

PCCM Entity

Validation of Performance
Improvement Projects

Required

State Discretion

Validation of Performance Measures

Required

Review of Compliance with Medicaid
Managed Care Regulations

Required

Validation of Network Adequacy

Required

N/A

States may choose to contract with different entities, including more than one EQRO, for different EQR-related activities.
For example, the state might validate performance improvement projects (see Protocol 1) itself, contract with EQRO A for
the validation of performance measures (see Protocol 2) and contract with EQRO B for the compliance review (see Protocol
3). Said state could then contract with EQRO A, B, or a third EQRO (C) to conduct the EQR and produce the EQR technical
report. For information on state contracting options for EQR, see 42 C.F.R. § 438.356 (as cross referenced at § 457.1250 for
CHIP.
13
Until the network adequacy validation protocol is issued, MCOs, PIHPs, and PAHPs will only be subject to three
mandatory EQR-related activities: Protocol 1 (Validation of PIPs), Protocol 2 (Validation of Performance Measures), and
Protocol 3 (Review of Compliance with Medicaid Managed Care Regulations) (Figure 4).
12

While regulations do not require PCCM entities to conduct PIPs as a part of their QAPI programs, states may choose to
require their PCCM entities to do so. States that require PCCM entities to conduct PIPs should consider validating those
PIPs.

INTRODUCTION | 11

MCP Type
EQR-Related Activity

MCO

PIHP

PAHP

PCCM Entity

Validation of Encounter Data
Reported by the MCP

State Discretion

Administration or Validation of Quality
of Care Surveys

State Discretion

Calculation of Additional Performance
Measures

State Discretion

Implementation of Additional
Performance Improvement Projects

State Discretion

Conducting Focus Studies of Health
Care Quality

State Discretion

Assist with Quality Rating of Medicaid
and CHIP MCOs, PIHPs, and PAHPsa

State Discretion

N/A

States may not claim for these EQR-related activities until the EQR protocol is issued.

For Medicaid programs, EQR (including the production of the EQR technical report) and EQRrelated activities performed on MCOs, as well as the production of the EQR technical report are
eligible for Federal financial participation (FFP) at a 75 percent match rate (1) when conducted
by a qualified EQRO and (2) when the EQR-related activities are completed using
methodologies consistent with the protocols contained within this document. 15,16 EQR-related
activities conducted on MCOs by an entity other than a qualified EQRO are eligible for the 50
percent match rate. 17 EQR (including the production of the EQR technical report) and EQRrelated activities conducted on PIHPs, PAHPs and PCCM entities are eligible for the 50 percent
match rate. 18 For CHIP, EQR and EQR-related activities are subject to the 10 percent
administrative cap as required by section 2105(c)(2)(A) of the Act, but a state is eligible to
receive the state’s enhanced CHIP FFP match rate for these activities, regardless of which
entity complete the activity.

See 42 C.F.R. §§ 433.15 and 438.370(a) and the July 10, 2016 CMCS Informational Bulletin (CIB), Federal Financial
Participation for Managed Care External Quality Review, available at https://www.medicaid.gov/federal-policyguidance/downloads/cib061016.pdf.
16
If the state or the state’s agent that is not an MCP conducts the EQR-related activity on an MCO, it would be eligible for
the 50 percent match rate. See 42 C.F.R. § 438.370(a)–(b). When information from a Medicare or private accreditation
review of an MCO is used to support one or more mandatory EQR-related activities in place of a Medicaid review, the
EQRO’s analysis of the MCO data as part of the EQR is eligible for FFP at the 75 percent rate. The accreditation activities
that produce the information are not eligible for the FFP.
17
See 42 C.F.R. §§ 433.15 and 438.370(b).
15

18
See 42 C.F.R. § 438.370(b). Note that this is a change from the previous regulations, under which the enhanced match
was available for EQR of PIHPs to the same extent as MCOs. For further explanation of the change, see discussion in the
Medicaid and CHIP Managed Care Final rule at 81 FR 27498, 27715-27716 available at https://www.gpo.gov/fdsys/pkg/FR2016-05-06/pdf/2016-09581.pdf.

12 | INTRODUCTION

OVERVIEW OF THE EQR PROTOCOLS
CMS is required to develop EQR protocols to
guide and support the annual EQR process. 19
The first set of protocols was issued in 2003 and
updated in 2012 (recall Figure 1). CMS revised
the protocols in 2018 to incorporate regulatory
changes contained in the May 2016 Medicaid
and CHIP managed care final rule. The revised
protocols are also designed to improve the user
experience navigating through the components,
provide new tools to drive improvement using
current industry methodologies (such as rapid
cycle evaluation approaches), and offer
practical tips and best practices for reporting
(See box, Content of the Protocols).

Content of the Protocols
Each protocol describes:
• Purpose of the EQR-related activity
• How to conduct each activity within the protocol,
including:
○ Data sources and data collection activities to
promote data accuracy, validity, and reliability
○ Proposed method(s) for analyzing and
interpreting the data, and
○ Instructions, guidelines, worksheets, and/or
tools that may be used in implementing the
protocol

Figure 4 identifies the EQR protocols linked to each of the mandatory and optional EQR-related
activities, as well as the source of the regulations that guide the protocols.
In addition, an Information Systems Capabilities Assessment (ISCA) is a mandatory component
of the EQR as part of Protocols 1, 2, 3, and 4, as well as Protocols 5 and 7 (if applicable). Note
that the regulations at 42 C.F.R. §§ 438.350(e) and 457.1250(a) require that the information
provided to the EQRO for the annual EQR be obtained through methods consistent with these
EQR protocols. This standard applies to both the mandatory EQR-related activities and to any
optional EQR-related activities a state elects to apply to its MCPs. CHIP regulations additionally
require that each state that contracts with MCOs, PIHPs, or PAHPs must follow the EQR
requirements. 20

See section 1932(c)(2)(A)(iii) of the Social Security Act and 42 C.F.R. § 438.352.

See 42 C.F.R. § 457.1250(a) for information about the applicability of EQR to CHIP MCPs.

INTRODUCTION | 13

Figure 4. Overview of the EQR Protocols

Notes: All EQR-related activities apply to CHIP MCPs via 42 C.F.R. § 457.1250(a).
MCOs = managed care organizations; PAHPs = prepaid ambulatory health plans; PIPs = performance improvement projects; PIHPs
= prepaid inpatient health plans.

14 | INTRODUCTION

States must ensure that the privacy of patient information is protected in a manner consistent
with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) 21 throughout all
EQR-related activities and the EQR technical report process. Specifically, the final rule requires
that EQR technical reports do not disclose a patient’s identify or any Protected Health
Information (PHI). 22 Consistent with that obligation, states should ensure that their MCPs
comply with HIPAA and all other federal and state laws concerning confidentiality and
disclosure. The EQRO should ensure that its EQR-related data collection and reporting
activities meet these requirements.
The next section of this chapter discusses practical considerations for states before beginning
the EQR-related activities. The following section provides tips to guide the drafting of effective
EQR technical reports that document performance in regards to quality, timeliness, and access
to care; identify areas for improvement; and recommend interventions to improve the process
and outcomes of care. Links to the protocols and appendices are contained at the end of this
chapter. The four appendices are: Information Systems Capabilities Assessment (Appendix A),
Sampling Approaches for EQR Data Collection Activities (Appendix B), Acronyms Used in the
Protocols (Appendix C), and External Quality Review Glossary of Terms (Appendix D).

CONSIDERATIONS BEFORE CONDUCTING EQR-RELATED ACTIVITIES
Preparing to conduct EQR-related activities involves several steps (See box, Steps to Prepare
for EQR-Related Activities). EQR-related activities may be performed by the state; an agent of
the state that is not an MCP; or by an external quality review organization (EQRO). 23 These
protocols are applicable to EQR-related activities conducted by any of these entities. While most
states hire an EQRO to conduct the EQR-related activities, states may elect to conduct the
EQR-related activities themselves or to contract with an organization that is not an EQRO or an
agent that is not an MCP to perform these activities.
Steps to Prepare for EQR-Related Activities
1 Select an entity to conduct the EQR-related activity(ies)
○ Ensure that staff conducting EQR-related activities have the training and experience needed for the particular
activity(ies) they will be conducting
2 Provide clear, written understanding of the parameters of the review
○ List of MCPs for review
○ Select optional EQR-related activities (if applicable) in addition to the applicable mandatory EQR-related activities
○ Designate a timeframe for review
3 Review all applicable federal regulations, state regulations or standards, and MCP state contracts
4 Confirm with entity and all EQR participants
○ Each organization's responsibilities in collecting, reporting, and/or analyzing data
○ Which regulations, contracts, and/or initiatives should be evaluated
○ Which reviews will occur and tools used
○ A timeline identifying the start and completion of each protocol

21
22
23

See 42 C.F.R. §431 Subpart F and § 457.1110.
See 42 C.F.R. § 438.364(d).
See 42 CFR 438.358(a).

INTRODUCTION | 15

NONDUPLICATION FOR MANDATORY EQR-RELATED ACTVITIES
Nonduplication is intended to reduce administrative burden on MCPs and states while still
ensuring relevant information is available to EQROs for the annual EQR. The expansion of
nonduplication to three of the mandatory EQR-related activities (Protocols 1–3, Validation of
Performance Improvement Projects, Validation of Performance Measures, and Review of
Compliance with Medicaid Managed Care Regulations) 24 for all Medicaid managed care MCOs,
PIHPs, and PAHPs—not just those serving only dually eligible beneficiaries—provides
additional flexibility to states to reduce administrative burden. Nonduplication is an option for a
state only when the Medicare or accreditation review standards are comparable to the EQR
protocols (not vice versa). If a state elects to use nonduplication, it must document in its
managed care quality strategy the EQR-related activities for which it will utilize nonduplication
along with the state’s rationale for its determination that the Medicare or private accreditation
review standards are comparable to those in these protocols. 25 The federal requirements
related to nonduplication of mandatory activities are described in 42 C.F.R. § 438.360. Like
Medicaid, CHIP MCPs may submit information from a private accreditation review; however,
with regard to CHIP, information documenting compliance with Medicare Advantage standards
is not applicable as described in 42 C.F.R. § 457.1250(a).
Nonduplication allows a state to use information from a Medicare or private accreditation review
of an MCP in place of generating that information through one or more of three mandatory EQRrelated activities (Protocols 1–3, Validation of Performance Improvement Projects, Validation of
Performance Measures, and Review of Compliance with Medicaid Managed Care
Regulations). 26 To do so, the following conditions must be met:
•

The MCP is in compliance with the applicable Medicare Advantage or private accreditation
standards 27

•

The Medicare or private accreditation review standards are comparable to those
established through the EQR protocols for the three mandatory EQR-related activities

•

The MCP provides the state with all applicable reports, findings, and other results of the
Medicare or private accreditation review applicable to the specified EQR-related activities

The state is responsible for providing the EQRO with all information from the Medicare or
private accreditation review which is being used for nonduplication. The EQRO then assesses
the completeness of information from the accreditation review to determine the extent of
nonduplication, including confirming the comparable information fully meets the requirements for
completing the analysis and developing EQR findings and recommendations. If a state chooses
nonduplication, it must ensure the completion of any EQR-related activities (or components of
those activities) that are not addressed by the information from the Medicare or private
accreditation review. For example, if an accreditation review did not validate long term services
or supports (LTSS) or other non-Healthcare Effectiveness Data and Information Set (HEDIS®)

Nonduplication is not an option for the fourth mandatory EQR-related activity of network adequacy validation (42 C.F.R. §
438.358(b)(1)(iv)).
25
See 42 C.F.R. § 438.360(c) and 438.340(b)(10).
24

Prior to issuance of the Medicaid and CHIP final rule, such information could only be used to provide information which
would otherwise be gathered from performing the mandatory EQR-related compliance review.

See 42 C.F.R. § 422 subpart D.

16 | INTRODUCTION

measures required by the state as a part of an MCP’s QAPI program, that validation activity
would need to be completed for those measures.
It is important to note that even when information from a Medicare or private accreditation
review does not completely meet the requirements of an activity, that information can still be
used toward meeting the nonduplication requirements. For example, nonduplication might be
able to satisfy a subset of the regulatory requirements that are subjects of the compliance
review. In this example, the EQRO could use information from the nonduplication source for that
subset of requirements, and then the EQR-related activity would only need to be conducted on
the remaining requirements to fully assess compliance. Similarly, if a state requires its MCPs to
include 10 measures in QAPI and 5 are validated as a part of an accreditation review, only the
other 5 would need to be validated through the EQR-related activity. Validation information on
all 10 measures would then be provided to the EQRO for the EQR.
When information from a Medicare or private accreditation review of an MCP is used to support
one or more mandatory EQR-related activities, the EQRO’s analysis of the data is eligible for
FFP. The accreditation activities that produce the information are not eligible for the FFP. Note
that use of nonduplication is at the discretion of the state, not its MCPs.
What is the difference between nonduplication and exemption?
Nonduplication is a way to provide information for the annual EQR without conducting part of, or all of, one or more
EQR-related activities by using information yielded by a comparable review process. Under nonduplication, an MCO,
PIHP, or PAHP is still subject to EQR and will be included in the annual EQR technical report. Nonduplication may be
used at the state’s discretion and consistent with documentation in the state’s managed care quality strategy.
Exemption is an option which allows a state to exempt an MCO (but not a PIHP or PAHP) from the annual EQR
process under certain circumstances. If a state exempts an MCO from EQR, the MCO will not be included in the
annual EQR technical report. Exemption may be used at the state’s discretion when the following three conditions are
met:
• The MCO has both a current Medicare Advantage contract and a current Medicaid contract;
• The two contracts cover all or part of the same geographic area in the state; and
• The Medicaid contract has been in effect for at least two consecutive years before the exemption date, and during
those same two years, the MCO has been subject to EQR and met quality, timeliness, and access to health care
services standards for Medicaid beneficiaries
If a state wants to exempt an MCO from EQR, it must obtain either of the following:
• For MCOs reviewed by Medicare, the state must obtain annually the most recent Medicare review findings from the
MCO, including all data, correspondence, information, and findings relevant to the MCO’s compliance with
Medicare standards for (1) access, quality assessment and performance improvement, health services, or
delegation of these activities, (2) all measures of the MCO’s performance, and (3) results and findings of all
performance improvement projects for Medicare enrollees
• For MCOs reviewed by a private, national accrediting organization that CMS approves and recognizes for Medicare
Advantage Organization deeming, the state must require that the MCO provide a copy of all findings from its most
recent accreditation review if that review was used to meet certain requirements for Medicare external review, or to
determine compliance with Medicare requirements. At a minimum, findings must include accreditation review
results of evaluation of compliance with individual accreditation standards, any deficiencies, corrective action plans,
and summaries of unmet accreditation requirements
Complete requirements for exemption of MCOs are available at 42 C.F.R. § 438.362.

INTRODUCTION | 17

TIPS FOR DRAFTING EQR TECHNICAL REPORTS
A qualified EQRO 28 is the only entity which
may conduct the annual EQR, that is, the
analysis and evaluation of information
generated by the EQR-related activities (or
via nonduplication, if applicable) regarding
the quality, timeliness, and access to the
health care services that an MCP, or its
contractors, furnish to beneficiaries. The end
product of the EQR is an EQR technical
report, which must be drafted by said EQRO
for the state. 29 CMS has developed tips to
help EQROs produce a report that both
satisfies regulatory requirements 30 and
clearly and concisely indicates the methods
that were used, the results that were
achieved, and recommendations for future
actions (See box, Key Changes to the EQR
Process). Specifically, EQROs should
produce reports that:
•

Key Changes to the EQR Process
• An accrediting body may not serve as an EQRO for
a health plan it accredited within the previous 3 years
• Expanded EQR to PAHPs and PCCM entities
• Added two new EQR-related activities (network
adequacy validation and assistance with the quality
rating of Medicaid and CHIP MCOs, PIHPs, and
PAHPs)
• Clarified that information from the EQR-related
activities, conducted in a manner consistent with the
EQR protocols, must be used to complete the EQR
report
• States cannot substantively revise the EQR technical
report without evidence that errors occurred or that
key information was omitted
• States must finalize the report by April 30th of each
year and post it on the state’s website

Document procedures used to analyze the data collected and how the EQRO reached its
conclusions regarding the quality, timeliness, and access to care provided by the MCP. For
each EQR-related activity, the EQRO must identify:
○

The objectives

○

Technical methods for data collection and analysis

○

Description of the data obtained

○

Conclusions based on the data analysis

•

Assess each MCP's strengths and weaknesses individually, including quality, timeliness,
and access to health care services furnished to Medicaid and/or CHIP beneficiaries

•

Recommend improvements to the quality of health care services furnished by each MCP,
including how the state can target goals and objectives in the quality strategy to better
support improvement in the quality, timeliness, and access to health care services furnished
to Medicaid beneficiaries

•

Ensure methodologically appropriate, comparative information about all MCPs, consistent
with guidance provided in the protocols

•

Assess the degree to which each MCP has effectively addressed the recommendations for
quality improvement made by the EQRO during the previous year's EQR

28
29
30

See 42 C.F.R. § 438.354 for information about the competence and independence requirements for an EQRO.
See 42 C.F.R. § 438.364 (as cross referenced at §457.1250 for CHIP).
See 42 C.F.R. § 438.364 (as cross referenced at §457.1250 for CHIP).

18 | INTRODUCTION

The report must be available to the public (both upon request and on the state’s website) and
must assure the privacy of patient information.
To be of greatest use to states and other stakeholders, EQROs should draft reports that are
actionable, clear, and concise; that highlight substantive findings; and that contain actionable
recommendations (See box, Tips for Drafting an Effective EQR Technical Report, next page).
The EQRO should prepare an aggregate report that summarizes results across all MCPs and
provides state-level recommendations for performance improvement. Separate aggregate
reports can be provided by type of MCP if appropriate (for example, one aggregate report on all
of a state’s MCOs and a separate aggregate report on all of the state’s behavioral health
PIHPs).

INTRODUCTION | 19

Tips for Drafting an Effective EQR Technical Report
1 Use the names of MCPs when referring to plan performance
o Findings and comparisons should refer to MCPs by name in order to facilitate transparency and stakeholder
understanding of specific plan performance
2 Highlight substantive findings concerning the extent to which MCPs are furnishing high quality, timely, and
appropriate access to health care services
○ Findings should focus on the specific strengths and weaknesses that were identified, rather than on numerical
ratings or validation scores obtained under the EQRO’s review methodology
3 Contain specific recommendations for improvement of identified weaknesses
○ When weaknesses are identified, the report should include the EQRO’s understanding of why the weakness
exists and suggest steps for how the MCP—potentially in concert with the state—can best address the issue. If
the cause for the weakness is unclear or unknown, the EQRO should suggest how the MCP and/or state can
identify the cause
○ When determining recommendations, EQROs should consider whether the suggested actions are within the
authority of the MCP (or state)
4 Include assessments of MCPs’ responses to previous recommendations
○ While required, such assessments have historically been missing from some reports
○ EQROs should conduct and report out on this activity, and should document assessments with the same
specificity used when reporting on initial findings
5 Aggregate findings across plans, and show comparisons among the state’s plans
○ Providing this context makes it easier for stakeholders to understand the results of the review by providing
context for the findings concerning individual MCPs, and to more readily determine whether issues are localized
or systemic
6 EQROs should consider the merits of displaying previous recommendations, plan responses and actions,
and new recommendations in one chart
○ This enables a comprehensive view of the history of each MCP’s EQR reviews
○ The comparative information should include tables presenting, for all plans, performance measure scores, and
PIP ratings and scores
○ Charts can be used to display non-compliance with each of the reviewed state and federal standards
7 Aim for clarity and concise presentation
○ While every EQR review necessarily gathers and processes a substantial amount of material, non-essential
narrative makes it difficult for readers to identify the most relevant information
○ EQROs should attempt to limit the body of reports to less than 50 pages, and use tables to showcase key
findings
○ Because not all readers have deep experience in the areas covered by EQR, avoid technical language and
jargon when possible
○ To maximize interpretability of results, provide context for all statistics included in the report
○ To provide a comprehensive view of Medicaid and CHIP managed care quality, consider drafting an aggregate
report that includes all MCPs, or all of a specific type of MCP
○ To facilitate use of the reports for topic-specific analyses, submit a searchable PDF to enable stakeholders to
review topics of interest
8 Make report publicly available and comply with privacy protections
○ The report must be available to the public (both upon request and on the state’s website) and must assure the
privacy of patient information, consistent with the Health Insurance Portability and Accountability Act of 1996
(HIPAA) (42 C.F.R. §431 Subpart F and § 457.1110). States should ensure that their MCPs comply with HIPAA
and all other federal and state laws concerning confidentiality and disclosure. The EQRO should ensure that its
EQR-related data collection and reporting activities are consistent with these requirements

20 | INTRODUCTION

GETTING STARTED ON THE EQR PROTOCOLS
So far, this chapter has provided background on the mandatory and optional EQR-related
activities and the associated protocols, discussed considerations before conducting EQRrelated activities, provided an overview of nonduplication, and shared tips on drafting the EQR
technical report. Now it’s time to review the protocols for each of the activities and begin
planning the approach to conducting the EQR. Use the “Go Now!” buttons to navigate to the
individual protocols and the appendices.
It should be noted that the protocols in this document are designed to support the completion of
the EQR-related activities, which in turn assists the state meet the requirement to conduct an
EQR of its MCPs and for contracted EQROs to meet the requirements of producing an annual
EQR technical report. If a state prefers to use methods consistent with but not identical to these
protocols to conduct EQR-related activities, the state is encouraged to discuss the alternative
methods with CMS before implementation to assure the methods meet regulatory standards. If
you have any questions related to the EQR protocols or alternative methods, please contact
CMS via the TA mailbox, [email protected].

INTRODUCTION | 21

MANDATORY EQR-RELATED ACTIVITIES
Protocol 1 – Validation of Performance Improvement Projects

MCOs, PIHPs, and PAHPs are required to implement performance improvement
projects (PIPs) that focus on both clinical and non-clinical aspects of care. Protocol 1
specifies procedures for EQROs to use in assessing the validity and reliability of a PIP
(42 C.F.R. § 438.358(b)(i)).
Protocol 2 – Validation of Performance Measures

MCPs must report standard performance measures as specified by the state. The state
must provide to the EQRO and the MCP the performance measures to be calculated,
the specifications for the measures, and the state reporting requirements. 31 Protocol 2
tells the EQRO how to:
•

Evaluate the accuracy of the Medicaid/CHIP MCP reported performance measures
based on the measure specifications and state reporting requirements; and

•

Evaluate if the MCP followed the rules outlined by the state agency for calculating the
measures (42 C.F.R. § 438.358(b)(ii))

This protocol also applies when a state requires its MCPs to submit data to the state so
that the state can calculate the standard performance measures.
Protocol 3 – Review of Compliance with Medicaid and CHIP
Managed Care Regulations

The EQR is required to include a compliance review of each MCP once in a 3-year period.
Protocol 3 specifies procedures to determine the extent to which MCPs comply with standards
set forth at 42 C.F.R. § 438.358(b)(iii), state standards, and MCP contract requirements.
Note that states may meet the 3-year requirement in different ways: for example, some review all
MCPs at the same time once every 3 years; others conduct a complete compliance review on a
subset of plans each year on a 3-year cycle. While a full compliance review is only required for
each MCP once every 3 years, the state must address any EQR findings in the next reporting year.
Protocol 4 – Validation of Network Adequacy

[Reserved]
As noted earlier, a key step to getting started is for the state to determine what entity will conduct the EQR-related activities. EQRrelated activities may be performed by the state; an agent of the state that is not an MCP, PIHP, PAHP, or PCCM entity; or by an
external quality review organization (EQRO).The protocols are applicable to EQR-related activities conducted by any of these entities.
While most states hire an EQRO to conduct the EQR-related activities, the state may elect to conduct the EQR-related activities
themselves or to contract with an EQRO or an agent that is not an MCP to perform these activities. For the purposes of the protocols
and ease of explanation, we refer to EQROs as the entity conducting the EQR-related activities. An EQRO is the only entity which
may conduct the annual EQR, that is, the analysis and evaluation of information generated by the EQR-related activities (or via
nonduplication, if applicable) regarding the quality, timeliness, and access to the health care services that an MCP, or its contractors,
furnish to beneficiaries. The end product of the EQR is an EQR technical report, which must be drafted by said EQRO.

22 | INTRODUCTION

OPTIONAL EQR-RELATED ACTIVITIES
Protocol 5 – Validation of Encounter Data Reported by the MCP

The states use managed care encounter data, which are data about a distinct service
provided to an enrollee, to better understand the health services delivered by the MCP,
assess and review quality, monitor program integrity, and determine capitation
payment rates. Protocol 5 specifies procedures for assessing the completeness and
accuracy of encounter data submitted by MCPs to the state. It also assists in the
improvement of processes associated with the collection and submission of encounter
data from MCPs to the state.
Protocol 6 – Administration or Validation of Quality of Care Surveys

Surveys are a common method of measuring health care quality, especially consumer
experience with care. Protocol 6 specifies procedures for conducting various types of
surveys and validating those surveys.
Protocol 7 – Calculation of Additional Performance Measures

The state uses performance measures to monitor the performance of MCPs over time, to
understand the MCPs’ impact on the Medicaid population, to compare the performance
of different MCPs, and to inform the selection and evaluation of quality improvement
activities. Protocol 7 specifies procedures for calculating MCP performance measures in
accordance with the state specifications. It also supplies information to the state on the
extent to which the MCP’s information system provides accurate and complete
information necessary for the calculation of performance measures.
Protocol 8 – Implementation of Additional Performance Improvement Projects

The state may conduct—or request an EQRO conduct—a PIP in addition to those
MCOs, PIHPs, and PAHPs are required to conduct as a part of their QAPI programs.
Protocol 8 specifies procedures for implementing additional PIPs.
Protocol 9 – Conducting Focus Studies of Health Care Quality

The state may choose to conduct a study on a particular aspect of clinical and/or nonclinical services provided by its MCPs. Protocol 9 specifies procedures to plan and
carry out a focus study.
Protocol 10 – Assist with the Quality Rating of Medicaid and CHIP MCOs, PIHPs,
and PAHPs

[Reserved]

INTRODUCTION | 23

APPENDICES
The EQR protocol package includes four appendices to supplement information contained in the
protocols. Use the “Go Now!” buttons to navigate to the appendices.
Appendix A. Information Systems Capabilities Assessment

Protocols 1, 2, 3, 4, 32 5, and 7 require each state to assess their MCPs’ information
system (IS) capabilities. The regulations at 42 C.F.R. § 438.242 and 457.1233(d) also
require the state to ensure that each MCP maintains a health information system that
collects, analyzes, integrates, and reports data for areas including, but not limited to,
utilization, grievances and appeals, and disenrollments for reasons other than the loss
of Medicaid eligibility. Portions of the Information Systems Capabilities Assessment
(ISCA) are voluntary; however, there are components that relate directly to the
mandatory EQR-related activity protocols. It defines the recommended capabilities of
an MCP’s information system to meet the above noted regulatory requirements, as well
as how to assess the strength of the MCP’s information system capabilities. It includes
an overview of the processes for collecting, processing, and reporting data, and
guidance for:
•

Completing the ISCA assessment (by MCPs)

•

Reviewing ISCA and accompanying documents

•

Interviewing MCP staff, and

•

Analyzing ISCA findings

Appendix B. Sampling Approaches for EQR Data Collection Activities

This appendix provides an overview of sampling approaches that can be used in
Protocols 1, 2, 5, 6, 7, 8, and 9.
Appendix C. Acronyms Used in the Protocols

This appendix defines acronyms used in the Protocols.

Appendix D. External Quality Review Glossary of Terms

This appendix defines terms used in the Protocols.

Protocol 4 is currently reserved. Once available, the ISCA will also be applicable to this protocol.

24 | INTRODUCTION

FOR FURTHER INFORMATION
Technical assistance resources related to EQR, including the EQR protocols, are available on
Medicaid.gov at https://www.medicaid.gov/medicaid/quality-of-care/medicaid-managedcare/external-quality-review/index.html.
Please submit any questions or requests for technical assistance related to EQR to
[email protected].

INTRODUCTION | 25

This page left blank for double-sided copying.

26 | INTRODUCTION

PROTOCOL 1. VALIDATION OF
PERFORMANCE IMPROVEMENT PROJECTS
A MANDATORY EQR-RELATED ACTIVITY

ACTIVITY 1: ASSESS THE PIP METHODOLOGY

ACTIVITY 3: VERIFY PIP FINDINGS (OPTIONAL)

BACKGROUND
States must require their Medicaid and
What is a PIP?
CHIP managed care plans (MCPs) to
conduct performance improvement
A PIP is a project conducted by
the MCP that is designed to
projects (PIPs) that focus on both
achieve significant improvement,
clinical and nonclinical areas each year
sustained over time, in health
as a part of the plan’s quality
outcomes and enrollee satisfaction.
assessment and performance
A PIP may be designed to change
improvement (QAPI) program, per 42
behavior at a member, provider,
C.F.R. §§ 438.330 and 457.1240(b)
and/or MCP/system level.
(See box, What is a PIP?). 33 This
This protocol is used to verify that
external quality review (EQR)-related
a PIP used sound methodology in
activity validates the PIPs that the MCP
its design, implementation,
was required to conduct as part of its
analysis, and reporting.
QAPI program. The external quality
review organization (EQRO) reviews
the PIP design and implementation using documents provided by the
MCP, which may be supplemented with interviews of MCP staff. The
EQRO then reports to the state on its findings from reviewing and
validating the PIP(s) in the EQR technical report. As noted in the
Introduction, states have the option to use information from a Medicare
or private accreditation review of an MCP to provide information for the
annual EQR instead of conducting this mandatory EQR-related activity. 34, 35

PROTOCOL ONE

ACTIVITY 2: PERFORM OVERALL VALIDATION AND REPORTING OF
PIP RESULTS

33
At a minimum, a single PIP that focuses on both clinical and non-clinical aspects of care may satisfy
this requirement. Otherwise, a state must require at least two PIPs, one clinical and one non-clinical.
34
If the state elects to use nonduplication for this mandatory EQR-related activity (42 C.F.R. §
438.360, Nonduplication of mandatory activities with Medicare or accreditation review), then the state
must ensure that all information from the Medicare or private accreditation review is provided to the
EQRO for analysis and inclusion in the annual EQR technical report. (See 42 C.F.R. §
438.360(a)(1)–(3) for additional details regarding the circumstance under which nonduplication is an
option). Use of nonduplication must be identified in the state’s quality strategy (see 42 C.F.R. §
438.360(c) and 438.340(b)(10)). CHIP cross-references to this requirement at §457.1250, but does
not allow for the use of Medicare review activities for the purposes of nonduplication.

A state may not utilize nonduplication if Medicare has accepted an only attestation of a plan’s
QIP. In the context of this EQR-related activity, the QIP would have to undergo validation as part
of a Medicare review in order for nonduplication to be an option. See 42 C.F.R. § 438.360(a)(2).

A related protocol, Protocol 8. Implementation of Additional Performance Improvement Projects,
specifies procedures for implementation of additional PIPs in accordance with state
specifications.

GETTING STARTED ON PROTOCOL 1
To complete this protocol, the EQRO undertakes two required activities and one optional activity
for validating the PIPs for each MCP (Figure 1.1).
Figure 1.1. Protocol 1 Activities

Two supplemental resources are available to help EQROs validate performance improvement
projects, including:
•

Worksheets for Protocol 1. PIP Validation Tools and Reporting Framework, a set of
worksheets that can be used to guide and record answers for the validation of PIPs and
reporting of summary PIP information, based on activities 1 through 3 and associated steps
in this protocol

•

Appendix B. Sampling Approaches for EQR Data Collection Activities, which provides an
overview of sampling methods that could be used in this protocol

The remainder of this protocol outlines the steps associated with Activities 1 through 3.

28 | PROTOCOL ONE

ACTIVITY 1: ASSESS THE PIP METHODOLOGY
The EQRO should complete the nine steps in Activity 1, listed below, and answer the questions
posed in each step.
Step 1: Review the Selected PIP Topic

WORKSHEET 1.1

PIP topics should target improvement in relevant areas
of clinical and non-clinical services. In this step, the
EQRO determines the appropriateness of the selected
PIP topic(s). It is recommended that the aims of the
National Quality Strategy be considered when
developing PIP topics:
•

Better care for patients and families

•

Improved health for communities and populations

•

Affordable health care

Resources for Activity 1, Step 1
Worksheet 1.1. Review the Selected PIP
Topic
• Template for assessing the
appropriateness of the PIP topic,
including how the PIP topic was
selected, the consideration of the CMS
Child and Adult Core Set measures,
and input from enrollees or providers

More information about the National Quality Strategy is available from the Agency for
Healthcare Research and Quality (AHRQ) at
https://www.ahrq.gov/workingforquality/about/index.html.
CMS also suggests that PIP topics align with CMS-identified priorities:
•

More information about CMS priorities and initiatives is available on the CMS Medicaid
Quality of Care webpage at https://www.medicaid.gov/medicaid/quality-of-care/index.html

In addition, the state should review its performance on the CMS Core Set of Children’s Health
Care Quality Measures for Medicaid and the Children’s Health Insurance Program (CHIP) (the
Child Core Set) and the Core Set of Health Care Quality Measures for Adults Enrolled in
Medicaid (the Adult Core Set) measures to identify opportunities to improve performance on a
quality measure(s) through a managed care PIP.
Step 2: Review the PIP Aim Statement

WORKSHEET 1.2

In this step, the EQRO assesses the appropriateness and
adequacy of the aim statement. The PIP aim statement
identifies the focus of the PIP and establishes the
framework for data collection and analysis. The PIP aim
statement should define the improvement strategy,
population, and time period. It should be clear, concise,
measurable, and answerable (See Q&A box below). Table
1.1 provides a critique of illustrative PIP aim statements.

Resources for Activity 1, Step 2
Worksheet 1.2. Review the PIP Aim
Statement
• Template for reviewing the PIP Aim
Statement

PROTOCOL ONE | 29

Q: How do we know if a PIP aim statement is clear, concise, measurable, and answerable?
A: A PIP aim statement is clear, concise, measurable, and answerable if the statement specifies measureable
variables and analytics for a defined improvement strategy, population, and time period. Potential sources of
information to help form the PIP aim statement include:
○ State data relevant to the topic being studied
○ MCP data relevant to the topic being studied
○ CMS Child and Adult Core Set performance measures
○ Enrollee focus groups or surveys
○ Relevant clinical literature on recommended care and external benchmarks

Table 1.1. Critique of example PIP aim statements
Example PIP aim statements
Poor PIP
Aim
Statement

Does the MCP adequately address psychological
problems in patients recovering from myocardial
infarction?

Good PIP
Aim
Statement

Will the use of cognitive behavioral therapy in
patients with depression and obesity improve
depressive symptoms over a six-month period
during 2017?

Critique
• The PIP intervention is not specified
• It is unclear how impact will be measured
• The population and time period are not
clearly defined
• Specifies the PIP intervention (cognitive
behavioral therapy)
• Defines the population (patients with
depression and obesity) and time period
(six-month period during 2017)
• Specifies the measurable impact (improve
depressive symptoms)

Step 3. Review the Identified PIP Population

WORKSHEET 1.3

In this step, the EQRO assesses whether the MCP
Resources for Activity 1, Step 3
clearly identified the population for the PIP in relation to
Worksheet 1.3. Review the Identified
the PIP aim statement (such as age, length of
PIP Population
enrollment, diagnoses, procedures, and other
characteristics). Depending on the nature of the PIP aim
• Template for assessing whether the
PIP population was appropriately
statement, PIP population, and available data, the PIP
identified
may include the entire population or a sample of the
population. PIPs that rely on existing administrative data,
such as claims and encounter data, registry data, or vital
records, are typically based on the universe of the PIP population. PIPs that rely on either
medical record review or the hybrid method (which uses a combination of administrative data
and medical record review) typically include a representative sample of the identified population.
If a sample was used for the PIP, go to Step 4. If the entire population was studied, skip Step 4
and go to Step 5. If HEDIS® measures and sampling methodology are used, go to Step 5.

30 | PROTOCOL ONE

Step 4: Review the Sampling Method

WORKSHEET 1.4

In this step, the EQRO assesses the appropriateness of
the PIP’s sampling methods. Appropriate sampling
methods are necessary to ensure that the collection of
information produces valid and reliable results. Please
refer to Appendix B, Sampling Approaches for EQR
Data Collection Activities, for an overview of sampling
methodologies applicable to PIPs. When HEDIS®
measures are used and sampling is required (for
example, for measures calculated using the hybrid
method), HEDIS® sampling methodology should be used.

Resources for Activity 1, Step 4
Worksheet 1.4. Review the Sampling
Method
• Template for reviewing the suitability
of the sampling method based on the
PIP aim statement and population

Step 5: Review the Selected PIP Variables and Performance Measures

WORKSHEET 1.5

In this step, the EQRO assesses the variables selected
Resources for Activity 1, Step 5
for a PIP (See box, What is a Variable?). Variables in
PIPs can take a variety of forms as long as the selected
Worksheet 1.5. Review the Selected PIP
variables identify the MCP’s performance on the PIP
Variables and Performance Measures
questions objectively and reliably and use clearly defined
• Template for assessing the
indicators of performance. The PIP should include the
appropriateness of selected PIP
number and type of variables that are adequate to
variables and performance measures
answer the PIP question and for which appropriate and
reliable data are available to measure performance and
track improvement over time. Data availability should also be considered when selecting
variables for PIPs, as more frequent access to data, such as on a monthly, quarterly, or semiannual basis, supports continuous quality improvement (QI) and Plan Do Study Act (PDSA)
efforts and can allow an MCP or state to correct or revise course more quickly, if needed. If
plans collect monthly, quarterly, or semi-annual data, the plan should use a methodology to
ensure comparability, such as a rolling 12-month methodology. CMS encourages states to
select PIP variables and performance measures that can be examined on at least a semiannual basis. Variables used in PIPs may be continuous, categorical, or discrete (Table 1.2),
and use a variety of measurement scales to assess performance (Table 1.3).
What is a Variable?
A variable is a measurable characteristic, quality, trait, or attribute of a particular individual, object, or situation being
studied (see Table 1.2 for examples)
Tips for Choosing Variables for PIPs
When choosing variables, consider different types of variables and choose the variables that are best suited to the
available data, resources, and PIP aim statement

PROTOCOL ONE | 31

Table 1.2 Types of variables for PIPs
Variable type

Definition

Example

Continuous

Have a range of numerical values
Note: Data collected for a continuous variable can
be recoded as a discrete variable (e.g., an enrollee’s
blood pressure is above or below a specified level)

• Age, blood pressure, temperature,
height/weight, body mass index,
birthweight

Categorical

Have a range of non-ordered, qualitative values (or
categories)

• An enrollee survey question that asks
enrollees to identify the most
important among a list of incentives
offered to improve well-care visit rates

Discrete

Have a limited number of possible categories
Note: binary variables have two categories

• An enrollee has/has not received a flu
shot in the past 12 months

Table 1.3 Types of measurement scales for PIPs
Measurement
scales

Definition

Example

Interval

The distances between numbers denote significant
and interpretable differences (e.g., dollars, degrees,
inches, pounds) and the differences are
interpretable as higher or lower.

• The interval between an annual
income of $40,000 and $30,000 =
$10,000

Ordinal

Can be treated as quantitative in some
circumstances, and qualitative in others

• An enrollee survey question that asks
enrollees to rank their experience of
care on a scale from 1 (low quality) to
5 (high quality)

Nominal

The set of categories for a qualitative variable

• Mode of transportation to work (car,
bus, subway, bicycle, walk)

To the extent possible, CMS encourages MCPs to choose variables for PIPs that reflect health
outcomes. Performance measures are then used to measure these outcomes. For this protocol,
performance measures are used to monitor the performance of individual MCPs at a point in
time, to track MCP performance over time, to compare performance among MCPs, and to
inform the selection and evaluation of quality improvement activities. In addition, for the purpose
of this protocol, “outcomes” are defined as changes in patient health, functional status,
satisfaction, or goal achievement that result from health care or supportive services. 36 For
example, measures of avoidable hospitalizations or emergency department visits can
demonstrate the adequacy of access to preventive and primary care and effectiveness of care
for acute and chronic conditions. CMS recognizes that standardized performance measures
addressing outcomes may be limited because of the lag in observing changes in population
health relative to the timeframe for the PIP measurement period. Moreover, health outcomes
may be influenced by factors outside of the organization’s control, such as poverty, genetics,
and environmental factors. For these reasons, PIP outcomes do not always need to be health
outcomes per se, but should be linked to health outcomes. Figure 1.2 provides guidance for
selecting PIP performance measures for tracking performance and improvement in outcomes
over time.

See 42 C.F.R. § 438.320.

32 | PROTOCOL ONE

Figure 1.2. Guidance for selecting PIP performance measures

Objective and measurable
PIP performance
measures
should be:

Clearly defined
Based on current clinical knowledge or health services research
Collected at least semi-annually
Enrollee outcomes (for example, health or functional status), or
indicators of outcomes (avoidable hospitalizations, emergency
department use, enrollee satisfaction/experience)

The number and
complexity of PIP
performance
measures may
depend on:

The study question
The complexity of existing practice guidelines for a clinical
condition
Availability of data and resources to gather data

Potential sources
of supporting
information:

Clinical and non-clinical practice guidelines
Administrative data
(such as claims/encounters, registries, vital records)
Medical records

When selecting performance measures for a PIP, the MCP should first consider existing
measures because the specifications for these measures often have been refined over time,
may reflect current clinical guidance, and may have benchmarks for assessing MCP
performance. CMS encourages use of the CMS Child and Adult Core Set, Core Quality
Measure Collaborative, and certified community behavioral health clinics (CCBHC) measures. 37
Additional examples of existing measures include NCQA’s Healthcare Effectiveness Data
Information Set (HEDIS®) or measures that have been developed by AHRQ (such as the
prevention quality indicators, inpatient quality indicators, patient safety indicators, and pediatric
quality indicators). 38
When there are gaps in existing measures, the MCP may develop new measures based on
current clinical practice guidelines or health services research. The MCP should consider the
following questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study
question?

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityMeasures/Core-Measures.html.More information about measures for behavioral health
clinics is available at https://www.samhsa.gov/section-223/quality-measures.
38
More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.
37

PROTOCOL ONE | 33

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the MCP to reliably and accurately calculate the
measure? Are there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

Step 6: Review the Data Collection Procedures

WORKSHEET 1.6

In this step, the EQRO assesses the validity and
reliability of the procedures the MCP used to collect the
data that inform the PIP measurements. Validity means
that the data are measuring what is intended to be
measured. Reliability means that the data are producing
consistent results.
To ensure validity and reliability of the data collected as
part of the PIP, the MCP should develop a data
collection plan that specifies:
•

The data sources for the PIP

•

The data to be collected

•

How and when the data are to be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments used to collect the data

Resources for Activity 1, Step 6
Worksheet 1.6. Review the Data
Collection Procedures
• Template for reviewing the
appropriateness of PIP variables and
performance measures to track
improvement
• Includes an assessment of data
collection procedures overall and for
administrative and medical record
review

This step may involve two main kinds of data collection: administrative data sources and
medical record review. Procedures to collect data from administrative data systems will be
different from procedures for visual inspection of medical records or other primary source
documents. However, both types of data collection require assurances that data are valid and
reliable. CMS encourages states to utilize data sources that they are able to collect data from on
a regular basis (e.g., monthly, quarterly, and semi-annually).
•

Administrative data collection. Evaluating an administrative data collection methodology
emphasizes the system that stores the data and should focus on an estimation of the
degree of completeness of the administrative data used to measure performance and track
improvement. See Section 2 of Worksheet 1.6 for a checklist of administrative data
assessment questions. In addition, refer to Protocol 5, Validation of Encounter Data
Reported by the Managed Care Plan for more information on assuring the validity and
reliability of encounter data.

•

Medical record review. For some variables, medical record review may be the only valid
and reliable source of data. (Note that medical records may include other sources besides
the individual patient medical record, such as clinical tracking logs, manual registries, case
management records, and the like.) If the PIP requires medical record reviews, special

34 | PROTOCOL ONE

attention should be given to the qualifications of the medical record reviewers, the
specificity of the guidelines for data collection, and plans for ensuring inter- and intra-rater
reliability. The reviewers should have a standard protocol for reviewing records, have the
knowledge to interpret the records, and have been trained to identify and code the
information in the records using consistent decision rules. See Section 3 of Worksheet 1.6
for a checklist of medical record review assessment questions.
•

Hybrid data collection. The hybrid method uses both administrative and medical record
data. The hybrid method, when available, should be used when administrative data or
electronic health record (EHR) data are incomplete or may be of poor quality, or the data
elements for the measure are not captured in administrative data.

Step 7: Review Data Analysis and Interpretation of PIP Results

WORKSHEET 1.7

In this step, the EQRO assesses the quality of the data
Resources for Activity 1, Step 7
analysis and interpretation of PIP results. The review
assesses whether the appropriate techniques were
Worksheet 1.7. Review Data Analysis
and
Interpretation of PIP Results
used, and if the analysis and interpretation was accurate.
In addition, analysis and interpretation of the PIP data
• Template for assessing the quality
should be based on a continuous quality improvement
and completeness of the analysis
philosophy 39 and reflect an understanding of lessons
learned and opportunities for improvement. Interpretation
of the PIP results should involve assessing the causes of less-than-optimal performance and
collecting data to support the assessment.
Accurate data analysis is essential because the state or MCP may implement changes based
on the results. The primary source for the assessment should be analytic reports of PIP results
prepared by the MCP, including both baseline and repeat measurements of PIP outcomes. In
addition, the EQRO may assess the reasonableness of individual MCP results in relation to
existing state-level data, data from other MCPs, or industry benchmarks.
This protocol requires the analysis to assess the extent to which any change in performance is
statistically significant; however, it does not specify a level of statistical significance that must be
met. MCPs should indicate the level of statistical significance used in the analysis and which
findings were statistically significant.

Continuous Quality Improvement (CQI) refers to the ongoing study of processes to (1) improve services or outcomes, and
(2) prevent or minimize the chance of adverse outcomes. To do so, the organization identifies areas for improvement and
tests approaches.

PROTOCOL ONE | 35

Step 8: Assess the Improvement Strategies

WORKSHEET 1.8

In this step, the EQRO assesses the appropriateness of the interventions for achieving
improvement. This assessment builds on information gathered in Step 7 about the data analysis
and interpretation of PIP results. Significant, sustained improvement results from developing
and implementing effective improvement strategies (including strategies that are culturally and
linguistically appropriate for the target population). Selected strategies should be evidencebased, that is, there should be existing evidence (published or unpublished) suggesting that the
test of change would be likely to lead to the desired improvement in processes or outcomes (as
measured by the variables). The effectiveness of the improvement strategy is determined by
measuring change in performance according to the predefined measures that were selected in
Step 5.
A common approach used to guide improvement work is the Institute for Healthcare
Improvement’s (IHI) Model for Improvement. After using this model to define the parameters for
the improvement effort, the state then may test changes on a small scale using Plan Do Study
Act (PDSA) cycles (See Q&A box, What is the Model for Improvement and a PDSA cycle?).
PDSA cycles provide a methodology to test changes on a small scale and to apply rapid-cycle
learning principles to adjust intervention strategies over the course of the improvement. This
approach involves a continuous cycle of measuring and analyzing performance and requires
frequent review and adjustment. Data are evaluated on a regular basis and interventions are
then adapted based on what was learned. Interventions can then be scaled to larger settings or
populations if found effective. PIPs based on the Model for Improvement and PDSA process are
sometimes known as rapid-cycle PIPs.
Q: What is the IHI Model for Improvement?
A: This model provides a framework for conducting improvement work, which asks three questions:
○ What is your aim, and by when do you want to accomplish the aim?
○ How will you know that a change is an improvement?
○ What changes can you put in place to achieve your aim?
A Plan Do Study Act (PDSA) cycle is then used to structure the actual testing
Q: What is a PDSA cycle?
A: The steps in the PDSA cycle are to:
○ Plan: Plan the test or observation, including a plan for collecting data, and interpreting results
○ Do. Try out the test on a small scale
○ Study. Set aside time to analyze the data and assess the results
○ Act. Refine the change, based on what was learned from the test. Determine how to sustain the intervention, if
successful
This information about the Model for Improvement and PDSA approach was adapted from the Institute for Healthcare
Improvement, available at http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx.
An additional source of information is the Agency for Healthcare Research and Quality’s Health Care Innovations
Exchange, available at https://innovations.ahrq.gov/qualitytools/plan-do-study-act-pdsa-cycle.

36 | PROTOCOL ONE

Step 9: Assess the Likelihood that Significant and Sustained Improvement Occurred

WORKSHEET 1.9

In this step, the EQRO assesses the likelihood that
significant and sustained improvement occurred as a
result of the PIP. This assessment builds on findings
from the two previous steps (See box, Potential Sources
of Supporting Information).

Resources for Activity 1, Step 9
Worksheet 1.9. Assess the Likelihood
that Significant and Sustained
Improvement Occurred
• Template for assessing improvement

The EQRO should review the PIP methods and findings
as a result of the PIP
to assess whether there is evidence of statistically
significant improvement that may be associated with the
intervention implemented as part of the PIP. In addition, the EQRO may supplement the
quantitative assessment with information gathered through interviews with MCP staff and/or
providers about the implementation and results of the PIP intervention. Qualitative information
may inform the assessment of whether observed changes were likely to be attributable to the
PIP intervention, as opposed to a short-term event unrelated to the intervention or random
chance.
An important component of a PIP is to demonstrate sustained improvement. The EQRO should
assess whether repeated measurements were conducted, and if so, whether significant change
in performance relative to baseline measurement was observed. The repeat measurement
should use the same methodology as the baseline measurement. Any deviations in
methodology (such as sampling, data source, or variable definition) must be thoroughly
documented. If the PIP is in the early stages of implementation, and repeated measurements
are not yet available, the analysis plan should describe the methodology for subsequent
measurement. The EQRO should state in its final report which findings were found to be
significant either statistically, clinically, or programmatically over time.
Potential Sources of Supporting Information
• Statistical significance testing calculated on baseline and repeat indicator measurements (clarify that the
appropriate test was used, such as a t-test for small samples)
• Benchmarks for quality specified by the state Medicaid agency or found in industry standards
• Interviews with MCP staff and providers about the implementation and results of the PIP intervention

ACTIVITY 2: PERFORM OVERALL VALIDATION AND REPORTING OF PIP
RESULTS
WORKSHEET 1.10
WORKSHEET 1.11

PROTOCOL ONE | 37

In this activity, the EQRO assesses the overall validity
and reliability of the PIP methods and findings to
determine whether or not it has confidence in the
results. The EQRO will assign an overall validation
rating of high, moderate, low, or no confidence to the
PIP. The validation rating will be based on the
EQRO’s assessment of whether the MCP adhered to
acceptable methodology for all phases of design and
data collection, conducted accurate data analysis and
interpretation of PIP results, and produced significant
evidence of improvement.

Resources for Activity 2
Worksheet 1.10. Perform Overall Validation
of PIP Results
• Template to provide a validation rating
(high confidence, moderate confidence,
low confidence, or no confidence)
Worksheet 1.11. Framework for
Summarizing Information about PIPs
• Provides a structure for reporting general
information about the PIP, performance
measures and results, validation results,
and recommendations

To assign the overall validation rating, the EQRO will
review the assessments conducted as part of the nine
steps in Activity 1, and recorded in the Worksheets for
Protocol 1: PIP Validation Tools and Reporting Framework, or a similar tool. As studies always
have weaknesses, the EQRO will need to assess the relative strengths and weaknesses and
the extent to which they affect the confidence in the generalizability and usefulness of the PIP’s
findings. CMS suggests using the following validation rating to facilitate comparisons across
PIPs and across states: high confidence, moderate confidence, low confidence, and no
confidence.
The EQRO will report its findings to the state and the state will submit the final technical report
to CMS. The validation report should include a description of the PIPs that were validated and
the findings of the EQRO’s validation review. The EQRO and the state must include the actual
validation results of the PIPs in the final EQRO technical report for submission to CMS. The
EQRO is required to report the performance measurement data for the PIP validation in the
EQR technical report. 40 Please see “Tips for Drafting EQR Reports” in the Introduction for
further guidance to EQROs about how to produce a clear and concise report. In addition, please
see Worksheet 1.11. Framework for Reporting Summary PIP Information, for a suggested
format for summarizing PIP validation results in the EQR technical report.

ACTIVITY 3: VERIFY PIP FINDINGS (OPTIONAL)
A state may request that the EQRO verify the data produced by the MCP to determine if the
baseline and repeated measurements are accurate. While the validation of the PIP methodology
and findings is a mandatory activity, the verification of data or performance measures used in
the PIP is optional for EQROs. Verification activities can provide added confidence in reported
PIP results as they provide greater evidence that the findings are accurate.
However, verification is a resource-intensive activity that may not be necessary. For example, if
the PIP uses HEDIS® measures that have been certified by a third party, verification may not be
needed. Additionally, the Information System Capabilities Assessment (ISCA) may provide
assurances that the processes used to develop measures for the PIP are valid and reliable (See
Appendix A. Information System Capabilities Assessment). Similarly, if the PIP relies on
encounter data and the EQRO has conducted encounter data validation, the optional EQR-

See 42 C.F.R. § 438.364(a)(2)(iii). Cross-referenced by CHIP at 42 C.F.R. 457.1250.

38 | PROTOCOL ONE

related activity described in Protocol 5, further assurances may be provided about the accuracy
and completeness of the data used in the PIP.
If a state opts to have the EQRO verify the accuracy of the baseline and repeated
measurements, EQRO should focus on the processes through which data for the PIP were
obtained, processed, and analyzed. The verification process should begin with a thorough
review of existing resources:
•

Documentation produced by the MCP about the data, algorithms, and testing (e.g., code
reviews) related to the PIP data analysis

•

The assessment of the MCP’s information system produced as part of the ISCA

•

Any external validations of the accuracy and completeness of MCP encounter data (such
as the optional EQR-related activity)

•

Results of other EQR-related activities, such as performance measure validation or
compliance reviews

•

Results of private accreditation reviews or state Medicaid agency audits

In the event that no current assessment of an MCP’s information system or encounter data
exists, the state may choose to contract this function to assist in verifying the accuracy of the
PIPs.
Next, the EQRO should review specific algorithms and results related to the PIP measures.
Questions include:
•

Was the algorithm used to produce the PIP measures sound (that is, does the algorithm
measure what it is intended to measure, are the results consistent, and is the code well
documented)?

•

For measures calculated using administrative data: Did the MCP’s information system
capture enrollee information completely and accurately? To answer this question, the
EQRO may need to validate a sample of records to ensure the encounter data are
complete

•

For measures produced through medical record review: Did the MCP conduct a reabstraction of a small subset (validation sample) of the reviewed records to ensure the
abstraction was complete and accurate? Data retrieval and analysis should be conducted
on a small scale, with the validation sample following the same rules as the original PIP.

If validation of a sample of records is performed, the EQRO should perform statistical
correlations between the validation sample and the original PIP data. A variety of statistical
methods can be applied to assess the degree of correlation between the PIP and validation
measures. Two recommended methods are the Pearson correlation coefficient for continuous
data (e.g., age, income) and the Kappa statistic for categorical data (e.g., gender, race).
Assessing the algorithm together with the integrity of the data will provide a strong indication of
the accuracy of the PIP s findings.
END OF PROTOCOL 1

PROTOCOL ONE | 39

WORKSHEETS FOR PROTOCOL 1:
PIP VALIDATION TOOLS AND REPORTING FRAMEWORK
Instructions. Use these or similar worksheets to assist in validating Performance Improvement Projects (PIPs)
conducted by the managed care plan (MCP). These worksheets provide templates for validating PIPs and a
framework for reporting on validated PIPs in the external quality review (EQR) technical report. This tool includes the
following worksheets crosswalked to the applicable Activity and Step:
Worksheet name

Protocol activity and step

Worksheet 1.1. Review the PIP Topic

Activity 1. Step 1. Review the Selected PIP Topic

Worksheet 1.2. Review the PIP Aim Statement

Activity 1. Step. 2. Review the PIP Aim Statement

Worksheet 1.3. Review the Identified PIP Population

Activity 1. Step 3. Review the Identified PIP Population

Worksheet 1.4. Review the Sampling Method

Activity 1. Step 4. Review the Sampling Method

Worksheet 1.5. Review the Selected PIP Variables

Activity 1. Step 5. Review the Selected PIP Variables

Worksheet 1.6. Review the Data Collection Procedures

Activity 1. Step 6. Review the Data Collection
Procedures

Worksheet 1.7. Review Data Analysis and
Interpretation of PIP Results

Activity 1. Step 7. Review Data Analysis and
Interpretation of PIP Results

Worksheet 1.8. Assess the Improvement Strategies

Activity 1. Step 8. Assess the Improvement Strategies

Worksheet 1.9. Assess the Likelihood that Significant
and Sustained Improvement Occurred

Activity 1. Step 9. Assess the Likelihood that Significant
and Sustained Improvement Occurred

Worksheet 1.10. Perform Overall Validation of PIP
Results

Activity 2. Perform Overall Validation and Reporting of
PIP Results

Worksheet 1.11. Framework for Summarizing Information Activity 2. Perform Overall Validation and Reporting of
about Performance Improvement Projects (PIPs)
PIP Results
For each PIP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
EQRO interview date
Performance Improvement Project (PIP) name
PIP period date

MM/DD/YY to MM/DD/YY

Type of delivery system (check all that apply)

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM entity
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

Enrollees
# Medicaid/CHIP enrollees in MCP: ____
# Medicaid/CHIP enrollees in the PIP: ____
# Total number of MCP enrollees in the PIP: ____

Physicians
# MCP primary care physicians: _____
# MCP specialty physicians: _____

Note: IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care Organization;
PAHP = Prepaid Ambulatory Health Plan; PIHP = Prepaid Inpatient Health Plan; PCCM = Primary Case Management.

40 | PROTOCOL ONE

Worksheet 1.1. Review the Selected PIP Topic
PIP Topic ______________________________________________________________________________
Assess the appropriateness of the selected PIP topic by answering the following questions about the MCP and PIP.
Insert comments to explain “No” and “Not applicable (NA)” responses.
Question

Yes

Comments

1.1 Was the PIP topic selected through a
comprehensive analysis of MCP enrollee needs,
care, and services (e.g., consistent with
demographic characteristics and health risks,
prevalence of conditions, or the need for a specific
service by enrollees)? (If the PIP topic was
required by the state, please check “not applicable”
and note in comments.)
1.2 Did selection of the PIP topic consider
performance on the CMS Child and Adult Core Set
measures?
1.3 Did the selection of the PIP topic consider
input from enrollees or providers who are users of,
or concerned with, specific service areas? (If the
PIP topic was required by the state, please check
“not applicable” and note in comments.)
• To the extent feasible, input from enrollees who
are users of, or concerned with, specific services
areas should be obtained.
1.4 Did the PIP topic address care of special
populations or high priority services, such as:
• Children with special health care needs
• Adults with physical disabilities
• Children or adults with behavioral health issues
• People with intellectual and developmental
disabilities
• People with dual eligibility who use long-term
services and supports (LTSS)
• Preventive care
• Acute and chronic care
• High-volume or high-risk services
• Care received from specialized centers (e.g.,
burn, transplant, cardiac surgery)
• Continuity or coordination of care from multiple
providers and over multiple episodes
• Appeals and grievances
• Access to and availability of care
1.5 Did the PIP topic align with priority areas
identified by HHS and/or CMS?
1.6 Overall assessment: In the comments section,
note any recommendations for improving the PIP
topic.

PROTOCOL ONE | 41

Worksheet 1.2. Review the PIP Aim Statement
PIP Aim Statement ______________________________
Assess the appropriateness of the selected PIP topic by answering the following questions. Insert comments to
explain “No” and “Not Applicable (NA)” responses.
Question
2.1 Did the PIP aim statement
clearly specify the
improvement strategy,
population, and time period for
the PIP?
2.2 Did the PIP aim statement
clearly specify the population
for the PIP?
2.3 Did the PIP aim statement
clearly specify the time period
for the PIP?
2.4 Was the PIP aim statement
concise?
2.5 Was the PIP aim statement
answerable?
2.6 Was the PIP aim statement
measurable?
2.7 Overall assessment: In the
comments section, note any
recommendations for
improving the PIP aim
statement.

42 | PROTOCOL ONE

Yes

Comments

Worksheet 1.3. Review the Identified PIP Population
PIP Population _______________________________
Assess whether the PIP population was clearly identified by answering the following questions. Insert comments to
explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

3.1 Was the project population clearly
defined in terms of the identified PIP
question (e.g., age, length of the PIP
population’s enrollment, diagnoses,
procedures, other characteristics)?
• The required length of time will vary
depending on the PIP topic and
performance measures
3.2 Was the entire MCP population
included in the PIP?
3.3 If the entire population was included
in the PIP, did the data collection
approach capture all enrollees to whom
the PIP question applied?
• If data can be collected and analyzed
through an administrative data
system, it may be possible to study
the whole population. For more
guidance on administrative data
collection, see Worksheet 1.6.
3.4 Was a sample used? (If yes, use
Worksheet 1.4 to review sampling
methods).
• If the data will be collected manually
(such as through medical record
review), sampling may be necessary
3.5 Overall assessment: In the
comments section, note any
recommendations for identifying the
project population.

PROTOCOL ONE | 43

Worksheet 1.4. Review the Sampling Method
Overview of Sampling Method _________________________________________________
If HEDIS® sampling is used, check here, and skip the rest of this worksheet.
Assess whether the sampling method was appropriate by answering the following questions. Insert comments to
explain “No” and “Not Applicable (NA)” responses. Refer to Appendix B for an overview of sampling approaches for
EQR data collection activities.
Question
4.1 Did the sampling frame contain a
complete, recent, and accurate list of
the target PIP population?
• A sampling frame is the list from
which the sample is drawn. It
includes the universe of members of
the target PIP population, such as
individuals, caregivers, households,
encounters, providers, or other
population units that are eligible to
be included in the PIP. The
completeness, recency, and
accuracy of the sampling frame are
key to the representativeness of the
sample
4.2 Did the sampling method consider
and specify the true or estimated
frequency of the event, the confidence
interval to be used, and the acceptable
margin of error?
4.3 Did the sample contain a sufficient
number of enrollees taking into
account non-response?
4.4 Did the method assess the
representativeness of the sample
according to subgroups, such as those
defined by age, geographic location, or
health status?
4.5 Were valid sampling techniques
used to protect against bias? Specify
the type of sampling used in the
“comments” field.
4.6 Overall assessment: In the
comments section, note any
recommendations for improving the
sampling method.

44 | PROTOCOL ONE

Yes

Comments

Worksheet 1.5. Review the Selected PIP Variables and Performance Measures
Selected PIP Variables and Performance Measures:
_________________________________________________________________
Assess whether the selected PIP variables were appropriate for measuring performance and tracking improvement
by answering the following questions. Insert comments to explain “No” and “Not Applicable (NA)” responses.
Recall that CMS encourages MCPs to choose variables for PIPs that reflect health outcomes. Performance measures
are then used to measure these health outcomes. When selecting variables, the MCP should consider existing
performance measures.
Question

Yes

Comments

PIP variables
5.1 Were the variables adequate to answer the
PIP question?
• Did the PIP use objective, clearly defined,
time-specific variables (e.g., an event or
status that can be measured)?
• Were the variables available to measure
performance and track improvement over
time? (CMS encourages states to select
variables that can be examined on at least a
semi-annual basis
Performance measures
5.2 Did the performance measure assess an
important aspect of care that will make a
difference to enrollees’ health or functional
status?
5.3 Were the performance measures appropriate
based on the availability of data and resources
to collect the data (administrative data, medical
records, or other sources)?
5.4 Were the measures based on current clinical
knowledge or health services research?
• Examples may include:
○ Recommended procedures
○ Appropriate utilization (hospital admissions,
emergency department visits)
○ Adverse incidents (such as death,
avoidable readmission)
○ Referral patterns
○ Authorization requests
○ Appropriate medication use
5.5 Did the performance measures:
• Monitor the performance of MCPs at a point in
time?
• Track MCP performance over time?
• Compare performance among MCPs over
time?
• Inform the selection and evaluation of quality
improvement activities?

PROTOCOL ONE | 45

Question
5.6 Did the MCP consider existing measures,
such as CMS Child and Adult Core Set, Core
Quality Measure Collaborative, certified
community behavioral health clinics (CCBHC)
measures, HEDIS®, or AHRQ measures?
5.7 If there were gaps in existing measures, did
the MCP consider the following when developing
new measures based on current clinical practice
guidelines or health services research?
• Did the measure address accepted clinical
guidelines relevant to the PIP question?
• Did the measure address an important aspect
of care or operations that was meaningful to
MCP enrollees?
• Did available data sources allow the MCP to
reliably and accurately calculate the
measure?
• Were all criteria used in the measure defined
clearly (such as time periods, characteristics
of eligible enrollees, services to be assessed,
and exclusion criteria)?
5.8 Did the measures capture changes in
enrollee satisfaction or experience of care?
• Although enrollee satisfaction/experience is
an important outcome of care in clinical areas,
improvement in satisfaction should not be the
only measured outcome of a clinical project.
Some improvement in health or functional
status should also be addressed
• For projects in nonclinical areas (such as
addressing access or availability of services),
measurement of health or functional status is
preferred
5.9 Did the measures include a strategy to
ensure inter-rater reliability (if applicable)?
5.9 If process measures were used, is there
strong clinical evidence indicating that the
process being measured is meaningfully
associated with outcomes?
• This determination should be based on
published guidelines, including citations from
randomized clinical trials, case control
studies, or cohort studies
• At a minimum, the PIP should be able to
demonstrate a consensus among relevant
practitioners with expertise in the defined area
who attest to the importance of a given
process
5.10 Overall assessment: In the comments
section, note any recommendations for
improving the selected PIP variables and
performance measures.

46 | PROTOCOL ONE

Yes

Comments

Worksheet 1.6. Review the Data Collection Procedures
Assess whether the data collection procedures were valid and reliable by answering the following questions. This
worksheet includes three sections: (1) overall data collection procedures, (2) data collection procedures for
administrative data sources, and (3) data collection procedures for medical record review. Insert comments to explain
“No” and “Not Applicable (NA)” responses.

Section 1: Assessment of Overall Data Collection Procedures
Question

Yes

Comments

6.1 Did the PIP design specify a systematic
method for collecting valid and reliable data that
represents the population in the PIP?
6.2 Did the PIP design specify the frequency of
data collection? If yes, what was the frequency
(for example, semi-annually)?
6.3 Did the PIP design clearly specify the data
sources?
• Data sources may include:
○ Encounter and claims systems
○ Medical records
○ Case management or electronic visit
verification systems
○ Tracking logs
○ Surveys
○ Provider and/or enrollee interviews
6.4 Did the PIP design clearly define the data
elements to be collected?
• Accurate measurement depends on clear and
concise definitions of data elements (including
numerical definitions and units of measure)
6.5 Did the data collection plan link to the data
analysis plan to ensure that appropriate data
would be available for the PIP?
6.6 Did the data collection instruments allow for
consistent and accurate data collection over the
time periods studied?
6.7 If qualitative data collection methods were
used (such as interviews or focus groups), were
the methods well-defined and designed to collect
meaningful and useful information from
respondents?
6.8 Overall assessment: In the comments
section, note any recommendations for
improving the data collection procedures.
Note: Include assessment of data collection
procedures for administrative data sources and
medical record review noted below.

PROTOCOL ONE | 47

Section 2: Assessment of Data Collection Procedures for Administrative Data Sources
Question

Yes

Comments

6.9 If inpatient data was used, did the data
system capture all inpatient
admissions/discharges?
6.10 If primary care data was used, did primary
care providers submit encounter or utilization
data for all encounters?
6.11 If specialty care data was used, did
specialty care providers submit encounter or
utilization data for all encounters?
6.12 If ancillary data was used, did ancillary
service providers submit encounter or
utilization data for all services provided?
6.13 If LTSS data was used, were all relevant
LTSS provider services included (for example,
through encounter data, case management
systems, or electronic visit verification (EVV)
systems)?
6.14 If EHR data was used, were patient,
clinical, service, or quality metrics validated for
accuracy and completeness as well as
comparability across systems?

Section 3: Assessment of Data Collection Procedures for Medical Record Review
Question
6.15 Was a list of data collection personnel
and their relevant qualifications provided?
• Data collection personnel require the
conceptual and organizational skills to
abstract data. These skills will vary
depending on the nature of the data and
the degree of professional judgment
required. For example, trained medical
assistants or medical records clerks may
collect data if the abstraction involves
verifying the presence of a diagnostic test
report. However, experienced clinical staff
(such as registered nurses) should be
used to extract data to support a judgment
about whether clinical criteria are met
6.16 For medical record review, was interrater and intra-rater reliability described?
• The PIP should also consider and address
intra-rater reliability (i.e., reproducibility of
judgments by the same abstractor at a
different time)

48 | PROTOCOL ONE

Yes

Comments

Question

Yes

Comments

6.17 For medical record review, were
guidelines for obtaining and recording the
data developed?
• A glossary of terms for each project should
be developed before data collection begins
to ensure consistent interpretation among
and between data collection staff
• Data collection staff should have clear,
written instructions, including an overview
of the PIP, how to complete each section
of the form or instrument, and general
guidance on how to handle situations not
covered by the instructions. This is
particularly important when multiple
reviewers are collecting data

PROTOCOL ONE | 49

Worksheet 1.7. Review Data Analysis and Interpretation of PIP Results
Assess whether the data analysis and interpretation was appropriate by answering the following questions. Insert
comments to explain “No” and “Not Applicable” responses.
Question
7.1 Was the analysis conducted in
accordance with the data analysis plan?
7.2 Did the analysis include baseline and
repeat measurements of project outcomes?
7.3 Did the analysis assess the statistical
significance of any differences between the
initial and repeat measurements?
7.4 Did the analysis account for factors that
may influence the comparability of initial and
repeat measurements?
7.5 Did the analysis account for factors that
may threaten the internal or external validity
of the findings?
7.6 Did the PIP compare the results across
multiple entities, such as different patient
subgroups, provider sites, or MCPs?
• Comparing the performance across
multiple entities involves greater statistical
design and analytical considerations than
those required for a project assessing
performance of a single entity, such as an
MCP, over time
7.7 Were PIP results and findings presented
in a concise and easily understood manner?
7.8 To foster continuous quality
improvement, did the analysis and
interpretation of the PIP data include lessons
learned about less-than-optimal
performance?
• Analysis and interpretation of the PIP data
should be based on a continuous
improvement philosophy and reflect on
lessons learned and opportunities for
improvement
7.9 Overall assessment: In the comments
section, note any recommendations for
improving the analysis and interpretation of
PIP results

50 | PROTOCOL ONE

Yes

Comments

Worksheet 1.8. Assess the Improvement Strategies
Assess whether the selected improvement strategies were appropriate for achieving improvement by answering the
following questions. Insert comments to explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

8.1 Was the selected improvement strategy
evidence-based, that is, was there existing evidence
(published or unpublished) suggesting that the test
of change would be likely to lead to the desired
improvement in processes or outcomes (as
measured by the PIP variables)?
8.2 Was the strategy designed to address root
causes or barriers identified through data analysis
and quality improvement processes?
• Interventions that might have a short-term effect,
but that are unlikely to generate long-term change
(such as a one-time reminder letter to enrollees
or providers) are insufficient
• It is expected that interventions associated with
significant improvement will be system
interventions (such as educational efforts, policy
changes, or targeting of additional resources)
• It is expected that interventions should be
measurable on an ongoing basis (e.g., quarterly,
monthly) to monitor intervention progress
8.3 Was the rapid-cycle PDSA approach used to
test the selected improvement strategy?
• The steps in the PDSA cycle 41 are to:
○ Plan. Plan the test or observation, including a
plan for collecting data, and interpreting the
results
○ Do. Try out the test on a small scale
○ Study. Set aside time to analyze the data and
assess the results
○ Act. Refine the change, based on what was
learned from the test. Determine how to
sustain the intervention, if successful
• If tests of change were not successful (i.e., did
not achieve significant improvement), a process
to identify possible causes and implement
solutions should be identified
8.4 Was the strategy culturally and linguistically
appropriate? 42
8.5 Was the implementation of the strategy
designed to account or adjust for any major
confounding variables that could have an obvious
impact on PIP outcomes (e.g., patient risk factors,
Medicaid program changes, provider education,
clinic policies or practices)?

Institute for Healthcare Improvement: Science of Improvement, Testing Changes. Available at
http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx.
42
More information on culturally and linguistically appropriate services may be found at
41

http://minorityhealth.hhs.gov/templates/browse.aspx?lvl=2&lvlID=15.

PROTOCOL ONE | 51

Question
8.6 Building on the findings from the data analysis
and interpretation of PIP results (Step 7), did the
PIP assess the extent to which the improvement
strategy was successful and identify potential followup activities?
8.7 Overall assessment: In the comments section,
note any recommendations for improving the
implementation strategies.

52 | PROTOCOL ONE

Yes

Comments

Worksheet 1.9. Assess the Likelihood that Significant and Sustained Improvement
Occurred
Assess the likelihood that significant and sustained improvement occurred by answering the following questions.
Insert comments to explain “No” and “Not Applicable (NA)” responses.
Question

Yes

Comments

9.1 Was the same methodology used for baseline
and repeat measurements?
9.2 Was there any quantitative evidence of
improvement in processes or outcomes of care?
9.3 Was the reported improvement in performance
likely to be a result of the selected intervention?
• It is not necessary to demonstrate conclusively
(e.g., through controlled studies) that a change is
an effect of the intervention; it is sufficient to
show that the change might reasonably be
expected to result from the intervention
• It is not necessary to undertake data analysis to
correct for secular trends (e.g., changes that
reflect continuing growth or decline in a measure
because of external forces over an extended
period). The measured improvement should
reasonably be determined to have resulted from
the intervention
9.4 Is there statistical evidence (e.g., significance
tests) that any observed improvement is the result
of the intervention?
9.5 Was sustained improvement demonstrated
through repeated measurements over time?
9.6 Overall assessment: In the comments section,
note any recommendations for improving the
significance and sustainability of improvement as a
result of the PIP.

PROTOCOL ONE | 53

Worksheet 1.10. Perform Overall Validation of PIP Results
Provide an overall validation rating of the PIP results. The “validation rating” refers to the EQRO’s overall confidence
that the PIP adhered to acceptable methodology for all phases of design and data collection, conducted accurate
data analysis and interpretation of PIP results, and produced evidence of significant improvement. Insert comments
to explain the rating.
PIP Validation Rating (check one box)
High confidence
Moderate confidence
Low confidence
No confidence

54 | PROTOCOL ONE

Comments

Worksheet 1.11. Framework for Summarizing Information about Performance
Improvement Projects (PIPs)

1. General PIP Information
Managed Care Plan (MCP) Name:
PIP Title:
PIP Aim Statement:
Was the PIP state-mandated, collaborative, statewide, or plan choice? (check all that apply)
State-mandated (state required plans to conduct a PIP on this specific topic)
Collaborative (plans worked together during the planning or implementation phases)
Statewide (the PIP was conducted by all MCOs and/or PIHPs within the state)
Plan choice (state allowed the plan to identify the PIP topic)
Target age group (check one):
Children only (ages 0–17)*
Adults only (age 18 and over)
Both adults and children
*If PIP uses different age threshold for children, specify age range here:
Target population description, such as duals, LTSS or pregnant women (please specify):
Programs:

Medicaid (Title XIX) only

CHIP (Title XXI) only

Medicaid and CHIP

2. Improvement Strategies or Interventions (Changes tested in the PIP)
Member-focused interventions (member interventions are those aimed at changing member practices or
behaviors, such as financial or non-financial incentives, education, and outreach)

Provider-focused interventions (provider interventions are those aimed at changing provider practices or
behaviors, such as financial or non-financial incentives, education, and outreach)

MCP-focused interventions/System changes (MCP/system change interventions are aimed at changing MCP
operations; they may include new programs, practices, or infrastructure, such as new patient registries or data
tools)

3. Performance Measures and Results (Add rows as necessary)
Performance
measures
(be specific
and indicate
measure
steward and
NQF number
if
applicable):

Baseline
year

Baseline
sample
size and
rate

Most recent
remeasurement
year
(if applicable)
Not
applicable—PIP
is in planning or
implementation
phase, results not
available

Most recent
remeasurement
sample size and
rate
(if applicable)

Demonstrated
performance
improvement
(Yes/No)
Yes
No

Statistically
significant change
in performance
(Yes/No)
Specify P-value
Yes

Specify P-value:
<.01

<.05

Other (specify):

PROTOCOL ONE | 55

Performance
measures
(be specific
and indicate
measure
steward and
NQF number
if
applicable):

Baseline
year

Baseline
sample
size and
rate

Most recent
remeasurement
year
(if applicable)

Most recent
remeasurement
sample size and
rate
(if applicable)

Demonstrated
performance
improvement
(Yes/No)

Statistically
significant change
in performance
(Yes/No)
Specify P-value

Yes
Not
applicable—PIP
is in planning or
implementation
phase, results not
available

Yes

Specify P-value:
<.01
<.05
Other (specify):

Yes
Not
applicable—PIP
is in planning or
implementation
phase, results not
available

Yes

Specify P-value:
<.01
<.05
Other (specify):

4. PIP Validation Information
Was the PIP validated?

Yes

“Validated” means that the EQRO reviewed all relevant part of each PIP and made a determination as to its validity. In many
cases, this will involve calculating a score for each relevant stage of the PIP and providing feedback and recommendations.

Validation phase (check all that apply):
PIP submitted for approval
Planning phase
First remeasurement
Validation rating:

Second remeasurement

High confidence

Implementation phase

Baseline year

Other (specify):

Moderate confidence

Low confidence

No confidence

“Validation rating” refers to the EQRO’s overall confidence that the PIP adhered to acceptable methodology for all phases of
design and data collection, conducted accurate data analysis and interpretation of PIP results, and produced significant
evidence of improvement.

EQRO recommendations for improvement of PIP:

END OF WORKSHEETS FOR PROTOCOL 1

56 | PROTOCOL ONE

PROTOCOL 2. VALIDATION OF PERFORMANCE
MEASURES
A MANDATORY EQR-RELATED ACTIVITY

ACTIVITY 2: CONDUCT ONSITE VISIT ACTIVITIES
ACTIVITY 3: CONDUCT POST-ONSITE VISIT ACTIVITIES

BACKGROUND
States use performance measures to monitor the performance of individual
managed care plans (MCPs) at a point in time, to track performance over
time, to compare performance among MCPs, and to inform the selection
and evaluation of quality improvement activities. States specify standard
performance measures which the MCPs must include in their quality
assessment and performance improvement (QAPI) program. 43
In many cases, states and MCPs use measures included in the CMS
Core Set of Children’s Health Care Quality Measures for Medicaid and
the Children’s Health Insurance Program (CHIP) (the Child Core Set) and
the Core Set of Health Care Quality Measures for Adults Enrolled in
Medicaid (the Adult Core Set) to monitor and track quality of care in
Medicaid and CHIP. 44 While use of these measures by states is voluntary,
CMS encourages states to adopt and use the Child and Adult Core Set
measures to support their managed care quality measurement and
improvement initiatives. Many Core Set measures are part of the
Healthcare Effectiveness Data and Information Set (HEDIS®), and have
national and regional benchmarks.
Federal regulations at 42 C.F.R. § 438.330(c) require states to specify
standard performance measures for MCPs to include in their
comprehensive quality assessment and performance improvement (QAPI)
programs.45 Each year, the MCPs must: (1) measure and report to the
state the standard performance measures specified by the state; (2) submit
specified data to the state which enables the state to calculate the standard
performance measures; or (3) a combination of these approaches.

PROTOCOL TWO

ACTIVITY 1: CONDUCT PRE-ONSITE VISIT ACTIVITIES

More information about QAPI and performance measure validation is available at 42. C.F.R.
§ 438.330(b)(2) and (c), cross referenced by CHIP at §457.1240(b).
44
More information about the Child Core Set is available at https://www.medicaid.gov/
medicaid/quality-of-care/performance-measurement/child-core-set/index.html. More information
about the Adult Core Set is available at https://www.medicaid.gov/medicaid/quality-ofcare/performance-measurement/adult-core-set/index.html.
43

More information about QAPI and performance measures is available at 42. C.F.R. §
438.330(b)(2). This is cross-referenced by CHIP at 42 C.RF.R 457.1240(b).

This protocol is used to guide the validation of the performance measures specified by states for
inclusion in MCPs’ QAPI programs. It applies both when the QAPI performance measure is calculated
by the MCP and when it is calculated by the state. In general, the external quality review organization
(EQRO) must assess whether the performance measures calculated by the MCP46 are accurate
based on the measure specifications and state reporting requirements (42 C.F.R. § 438.330(b)(2)).
The state provides the list of performance measures to be validated, the specifications for the
measures, and the requirements for reporting. As noted in the Introduction, states have the option to
use information from a Medicare or private accreditation review of an MCP to provide information for
the annual EQR instead of conducting this mandatory EQR-related activity. 47,48 A related protocol,
Protocol 7. Calculation of Additional Performance Measures, may be used by EQROs to calculate
additional performance measures in accordance with state specifications.

GETTING STARTED ON PROTOCOL 2
Protocol 2 consists of three phases of activities: pre-onsite visit, onsite visit, and post-onsite visit (Figure
2.1). The activities take place before, during, and after the EQRO conducts an onsite visit with the MCP.
The validation process is interactive and concurrent with MCP performance measure calculation.
Figure 2.1. Protocol 2 Activities

While the protocol is written as if the MCP is calculating performance measures, the MCP may contract with another entity
to calculate and report on its behalf. Alternatively, 42. C.F.R. § 438.330(c)(ii), cross-referenced by CHIP at §457.1240(b),
allows the state to require the MCP to submit data to the state, which the state then uses to calculate the performance
measure. This protocol applies in either circumstance.

If the state elects to use nonduplication for this mandatory EQR-related activity (42 C.F.R. § 438.360 (Nonduplication of
mandatory activities with Medicare or accreditation review), then the state must ensure that all information from the Medicare or
private accreditation review is provided to the EQRO for analysis and inclusion in the annual EQR technical report. (See 42 C.F.R. §
438.360(a)(1)–(3) for additional details regarding the circumstance under which nonduplication is an option). Use of nonduplication
must be identified in the state’s quality strategy (see 42 C.F.R. § 438.360(c) and 438.340(b)(10)). CHIP cross-references to this
requirement at 42 C.F.R. § 457.1250, but does not allow for the use of Medicare review activities for the purposes of nonduplication.
47

A state may not utilize nonduplication if Medicare has accepted an only attestation of a plan’s QIP. In the context of this
EQR-related activity, the QIP would have to undergo validation as part of a Medicare review in order for nonduplication to be
an option. See 42 C.F.R. § 438.360(a)(2).

58 | PROTOCOL TWO

Two supplemental resources are available to help EQROs validate performance measures:
•

Worksheets for Protocol 2. Performance Measurement Validation Tools, which can be used
to prepare for and conduct pre-onsite, onsite, and post-onsite activities

•

Appendix A. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems

The remainder of this protocol outlines the steps associated with Activities 1 through 3.

ACTIVITY 1: CONDUCT PRE-ONSITE VISIT ACTIVITIES
Step 1: Define the Scope of the Validation

WORKSHEET 2.1
WORKSHEET 2.2

The performance measures each state requires will
depend on the specific needs of the state. The state
will provide the EQRO with a list of the performance
measures to be validated along with requirements for
data collection and reporting (e.g., sampling guidelines
and instructions for calculating numerators and
denominators).
The EQRO should use Worksheet 2.1. List of
Measures to be Validated to enumerate the
performance measures to be validated under Protocol
2, including their data source, reporting frequency, and
format. Five data sources are used to produce MCP
performance measures:

Resources for Activity 1, Step 1
Worksheet 2.1. List of Performance
Measures to be Validated
• Template for identifying the measures
the EQRO will validate for the state,
including the source, how frequently to
calculate each measure, and when
each measure is due to the state
Worksheet 2.2. Performance Measure
Validation Template
• Template for documenting audit
specifications for the validation
components of each performance
measure listed in Worksheet 2.1., and
to assess the MCP’s measurement and
reporting process for each component

Administrative data, such as claims/encounter
data, registries, or vital records

Medical record review

Administrative data supplemented by medical record review, referred to as the “hybrid”
method

Electronic health records

Surveys (survey administration and validation is addressed in Protocol 6)

PROTOCOL TWO | 59

For each of the measures to be validated, the EQRO should complete Worksheet 2.2.
Performance Measure Validation Template (or a similar tool). The worksheet is used to
systematically gather information about the validation components and audit specifications
based on existing documentation about the measure. Elements include:
•

Documentation related to the data collection and calculation method

•

Denominator calculation(s), including adequacy of the data sources to calculate the
denominator, operationalization of the measure-specific eligibility criteria, and adherence to
the measurement period

•

Numerator calculation(s), including adequacy of the data sources to calculate the
numerator, appropriateness of codes used to identify numerator compliance, avoidance of
double counting, and adherence to the measurement period

•

Sampling methodology (if used)

•

Reporting of rates and other supporting information, including documentation of deviations
(if any)

Worksheet 2.2 also contains an example of a completed performance measure validation
worksheet similar to what an EQRO would use before, during, and after its onsite visit. The
illustrative template is for Chlamydia Screening in Women (CHL-CH, Measure Steward:
National Committee for Quality Assurance, National Quality Forum (NQF) # 0033), which is
included in both the Child and Adult Core Sets and calculated using the administrative method.
During Activity 1, Step 1, the EQRO should begin to populate the audit specifications based on
the available measure documentation. Note that the worksheet is intended to serve as a “living
document” for the measure validation process and the EQRO can adapt the template if
necessary.
Step 2: Assess the Integrity of the MCP’s Information System

WORKSHEET A.1
WORKSHEET A.2

This step helps focus the onsite validation activities on aspects of the MCP’s information system
that are most likely to be an issue in the validation process. Before validating individual
performance measures, the EQRO must assess (1) the integrity of the MCP’s information
system, (2) the completeness and accuracy of the data produced, and (3) the readiness of the
MCPs’ data systems for calculating performance measures. As part of this step, the EQRO
conducts an Information Systems Capabilities Assessment (ISCA) for each MCP as described
in the following sections.

60 | PROTOCOL TWO

Conduct an Information Systems Capabilities Assessment
Before conducting the onsite visit, the EQRO should
provide the MCP with information on the ISCA
process, including Worksheet A.1 Information System
Capabilities Assessment Tool (See box, Resources to
Conduct an ISCA). The ISCA is used to validate MCP
information systems, processes, and data. The ISCA
corresponds to the objectives identified in this
protocol and addresses key components of
calculating performance measures including:
•

General information about the MCP

•

Membership/enrollment data systems

•

Claims/encounter data processing

•

Provider data

•

Data completeness

•

Integration of data for performance measure
calculation

Resources to Conduct an
Information Systems Capabilities
Assessment (ISCA)
The ISCA is used to validate MCP
information systems, processes, and data.
The ISCA provides a foundation for the
validation of performance measures.
• Appendix A explains how to conduct the
ISCA
• Worksheet A.1. ISCA Tool is completed
by the MCP and documents the
capabilities of the information systems,
processes, and data
• Worksheet A.2. ISCA Interview Guide
is used by EQROs to conduct follow-up
interviews with staff to record responses
and document specific issues based on
findings from Worksheet A.1

The ISCA provides information about the timing of any other recent, independent, documented
assessment such as a HEDIS Compliance Audit™. If the MCP recently had a comprehensive,
independent assessment of its information systems, the EQRO may review those results. If the
MCP has not had an ISCA within a timeframe determined by the state,49 the EQRO will conduct
an ISCA as part of this protocol. It is recommended that EQROs request that MCPs provide any
assessments of their IT systems conducted in the previous two years. The EQRO should
document the strengths and weaknesses of the MCP information system relevant to the types of
data used by the MCP in calculating performance measures. The EQRO should take into
account systems issues (such as missing data), when validating individual performance
measures and determining whether they are reportable.
Assess MCP Data Systems and Types
The EQRO should assess every data system and type of data the MCP processes to ensure the
required data are current and accurate, particularly at the time it extracts data for its
performance measures. The EQRO should assess changes in the MCP’s data systems that
might affect the production of the performance measures. Major changes, upgrades, or
consolidations within the system, or acquisitions/mergers with other MCPs may impact the
accuracy or completeness of required data elements. Elements that should be assessed for
each MCP data system and type include:
•

Membership/enrollment data

PROTOCOL TWO | 61

•

Provider data

•

Claims and encounter data

•

Medical records data

•

Pharmacy, laboratory, and other ancillary data

Membership/Enrollment Data
The EQRO should assess:
•

The MCP’s ability to track members over time, changes in enrollment, name changes, and
changes in coverage

•

The MCP’s processes to ensure membership/enrollment data are current and accurate

•

Changes in the MCP’s membership data systems that might affect the production of the
performance measures

•

Whether transactions between the MCP and state data systems (such as state eligibility
files) affect measure calculation through updating, correcting, or overwriting source data
(e.g., race or ethnicity information)

The EQRO should determine whether each MCP member is uniquely identifiable and can be
linked to the state’s Medicaid/CHIP eligibility file. The membership/enrollment database should
capture the following information for every member:
•

Unique member identifier (ID), including state-issued Medicaid/CHIP ID and CMS-issued
Medicare number (if applicable)

•

Eligibility category

•

Date of birth

•

Sex

•

Race and ethnicity

•

Primary language

•

Disability status

•

Enrollment and/or termination dates, including multiple enrollment and termination dates
within and across programs (preferably exact dates rather than monthly indicators)

•

Primary care provider (e.g., provider name, provider ID number, provider location)

Collecting and assessing membership and enrollment data is increasingly important due to the
quality strategy requirement for identifying, evaluating, and reducing health disparities based on
age, race, ethnicity, sex, primary language, and disability status (42 C.F.R. § 438.340(b)(6)).
Under this requirement, states must identify this demographic information for each Medicaid
enrollee and provide it to the MCP, PIHP, or PAHP at the time of enrollment.
In addition, to facilitate geographic stratification of performance (such as analyses of access and
timeliness of care), complete and accurate information on the household’s location of residence
(e.g., ZIP code) is also desirable.

62 | PROTOCOL TWO

Finally, to facilitate surveys of patient experience, complete contact information is essential. At a
minimum, name and address are required; phone numbers and email addresses are highly
desirable. See Protocol 6. Administration or Validation of Quality of Care Surveys for more
information.
Provider Data
The EQRO should review an MCP’s provider data system(s) to assess the MCP’s ability to track
providers over time, across multiple office locations, and through changes in participation. In
addition, the EQRO should assess how many contracted providers use electronic health records
(EHRs) and the extent to which EHRs are used in the calculation of an MCP’s performance
measures.
Claims and Encounter Data
Claims and encounter data should cover all types of services offered by the MCP and not
separately contracted by the state, such as hospital inpatient, hospital outpatient, primary care,
skilled nursing facility, nursing facility (custodial care), specialty care, behavioral health care,
family planning services, home health care, radiology, laboratory, pharmacy, dental care, and
vision care. The EQRO should note the following for each type of claim/encounter data
captured:
•

Total number of diagnosis and procedure codes (such as Healthcare Common Procedure
Coding System (HCPCS) codes and Current Procedural Terminology (CPT)® codes, the
American Dental Association’s Common Dental Terminology (CDT)© codes, and ICD-10
Procedure Coding System codes), captured by the system

•

Whether the principal diagnosis, secondary diagnoses, and procedure codes can be
accurately distinguished in the system

•

Maximum number of digits/characters captured for each data field in each type of claim or
encounter

The accuracy and validity of measures may be adversely affected if the information system
truncates codes or is unable to collect and/or differentiate among a sufficient number of codes.
The EQRO should understand the various coding systems and forms used by the MCP and its
vendors to capture and process clinical information through its claims and encounter databases.
The EQRO should assess how well the information system translates or maps these codes
back to the criteria for MCP performance measure reporting, and how it ensures the accuracy of
these translation processes.
The EQRO should also determine, through review of existing documentation or in consultation
with the MCP, whether certain diagnosis or procedure codes required for performance
measurement are not accurately or completely captured in the claims and encounter data
systems, such as maternity or dental care, behavioral health care, and preventive care services.
Medical Record Data
The EQRO should use medical record data to review:
•

Methods used to retrieve information from medical records

PROTOCOL TWO | 63

•

Training and tools that medical record review staff receive

•

Processes used to ensure accurate data retrieval, inter-rater reliability, and data entry into a
database used to produce performance measures

With increasing adoption of EHRs and state use of Health Information Exchanges, MCPs and
provider practices may use newer methods to extract information from the medical record. As
noted earlier, the EQRO should assess how electronic records are used in performance
measure calculation, and whether there are any special considerations in the validity and
reliability of these records for accurate measurement.
Pharmacy, Laboratory, and Other Ancillary Data
Pharmacy data use standardized codes for prescription drugs such as those promulgated by the
National Council for Prescription Drug Programs (NCPDP). 50 Laboratory services frequently use
a similar, nationally recognized system of coding (known as LOINC®). 51
Due to the diversity in the size, type, and ownership of pharmacy, laboratory, and other ancillary
providers, non-standard codes should be examined. When found, the EQRO should assess the
MCP’s system for cross-walking these different codes to store the necessary information in its
performance measure database. The EQRO should understand the MCP’s mapping system of
non-standard codes to standardized codes and the mechanism used to ensure the accuracy of
these translation processes.
If the MCP does not collect pharmacy, laboratory, or other ancillary data through an
administrative or claims database, it may retrieve these data from medical records. However,
medical records often are an unreliable source due to non-standard coding and terminology,
poor coordination of records, and insufficient record linkages between primary care and
specialist providers. These issues should be addressed during the claims/encounter data review
and the medical record review, and, if necessary, reflected on any corrective action plan.
The EQRO must assess the ability of the information system to link these different sources of
data. For example, to identify enrollees with diabetes, a MCP may need to combine diagnosis
code data from inpatient or ambulatory encounters (not all ongoing conditions are reported at
every encounter) with pharmacy data, lab data, and/or a disease registry, an MCP’s disease
management system, or a medical management system used by MCP staff, if one exists. Thus,
to determine whether enrollees with diabetes have received a retinal examination from an
ophthalmologist or optometrist within the previous year, the MCP would have to link diagnosis
and procedure code data from encounter forms, medical records, and/or claims data with
information about the specialty of the providers that performed the examinations for these
members.
Synthesis of Findings
The EQRO will review the findings from the ISCA across each of the data systems and types.
The EQRO should note any problem areas related to the adequacy of the MCP’s data systems
to calculate and report the required performance measures. Where a response is incomplete,

More information about NCPDP codes is available at https://www.ncpdp.org/.

More information about LOINC codes is available at https://loinc.org/.

64 | PROTOCOL TWO

indicates an inadequate process, or requires clarification, the EQRO should flag the issue for
follow-up and further review during the onsite activities.
Step 3: Conduct Detailed Review of Measures

WORKSHEET 2.2

The next step is to conduct a detailed review of measures, incorporating findings from the ISCA.
In its detailed review, the EQRO should identify measures that are most vulnerable to
inaccurate results based on its knowledge of the MCP’s data systems and processes. For
example, if the MCP uses global billing for maternity care, calculation of maternity measures
could be affected by the lack of separate claims for prenatal and postpartum care, and thus,
performance measurement results for such measures could be significantly under-reported.
Similarly, the EQRO should identify certain types of claims that may require linkage from other
data sources (such as laboratory, behavioral health, or dental) because the necessary codes
may not be available for all members.
The detailed review of each measure
involves a systematic assessment of the
code and output to assess adherence to
the specifications as well as the impact of
any systems issues on the accuracy and
completeness of the data (See box,
Resources for Detailed Review of
Measures). In addition, the EQRO should
pay special attention to frequentlyencountered issues in developing its audit
specifications based on findings from the
ISCA:

Resources for Detailed Review of Measures
A detailed review of each measure includes the following:
• Source code for the measure
• Data mapping, if applicable
• Measure workflow
• Data output at each stage of the measure calculation
• Record-level numerator and denominator data
Worksheet 2.2. Performance Measure Validation Template
• Template for documenting audit specifications for the
validation components of each performance measure
listed in Worksheet 2.1., and to assess the MCP’s
measurement and reporting process for each component

•

Claims-dependent denominators

•

Complex continuous enrollment
criteria

•

Use of global billing

•

Identification of live births (including linkage of mother and infant records)

•

Procedure codes that are infrequently billed by providers (such as developmental
screening, documentation of BMI, or BMI percentile in the medical record)

•

Ability to link claims and pharmacy data

•

Identification of practitioner type (especially mental health providers)

•

Multiple numerator events

•

Vendor-supplied data

During the detailed measure review, the EQRO should develop targeted audit specifications for
each measure to account for potential systems issues. The EQRO should record its audit
specifications and interim findings on Worksheet 2.2. Performance Measure Validation
Template, or a similar worksheet.
PROTOCOL TWO | 65

Review of HEDIS® Measures Calculated by HEDIS®-certified Software
If the state requires HEDIS® measures and the MCP used HEDIS®-certified software to calculate the measures, the
EQRO does not need to review source code for those measures. However, the EQRO is required to verify that the
measures were calculated as specified by the software and that systems issues did not compromise the accuracy and
completeness of the performance measures. As an example, when an MCP pays for prenatal and postpartum care as
part of a bundled maternity care payment, HEDIS® measures may be calculated according to the specifications but
the rates may be significantly under-reported using administrative data due to the lack of separate claims for prenatal
and postpartum care. Thus, the EQRO is required to review and validate the accuracy and completeness of HEDIS®
measures based on findings from the ISCA.

Step 4: Initiate Review of Medical Record Data Collection

WORKSHEET 2.3

The purpose of this step is to verify the accuracy of the
medical record review conducted by the MCP when
medical record data are used to calculate and report
performance measures. If a plan only used
administrative data, this step is not necessary.
To validate the integrity of the medical record review
processes, the EQRO conducts the validation in two
phases: the first phase assesses the initial
implementation of the process to allow corrections at an
early stage; the second phase is a retrospective review
of the accuracy of the medical record review abstraction
process.

Resources for Activity 1, Step 4
Worksheet 2.3. Medical Review
Validation Template
• Template provides instructions for
conducting the medical record review
and worksheets to summarize reabstraction findings from the review
(Worksheet 2.3, Table 1) and to
record the impact of findings from the
review (Worksheet 2.3, Table 2)

Review of Implementation of Medical Record Review
During the early implementation of the medical record abstraction process, the EQRO will
confirm the following about MCP activities:
1

Selection of staff with appropriate experience and credentials

Development of high-quality abstraction tools to collect the required information

Provision of effective staff training about the review process

Implementation of sound oversight procedures to assess reviewer performance (such as
validation of a sample of records or tests of inter-rater reliability)

The EQRO may review a convenience sample of records across measures to identify potential
problems for MCP correction. NCQA’s HEDIS Compliance Audit™ recommends selecting up to
10 difficult-to-review measures and obtain copies of at least 2 complete medical record review
tools and charts per measure. If the state requires fewer than 10 measures that rely on medical
record data, the EQRO should conduct the sample review for all medical record-dependent
measures. Completing this step early in the process allows the MCP to address identified issues
and resolve them during the initial stages of data collection.

66 | PROTOCOL TWO

Re-abstraction and Validation of Medical Record Review
The EQRO will conduct a retrospective medical record review for at least two measures that
include medical record review either alone or in combination with administrative data (known as
the hybrid method). The EQRO should target statistical validation to measures that are new,
complex, and dependent on the medical record data or those with previously identified issues.
For each measure, the EQRO will request a sample of 30 medical records with positive
numerator events and compare the completed abstraction information to the medical record to
determine the rate of agreement. If the agreement rate is less than 100 percent, the EQRO will
assess the degree of bias. Worksheet 2.3. Medical Record Review Validation Template
provides a detailed description of the medical record review process and validation tool. The
EQRO should summarize findings for the MCP from the medical record review validation,
including error rates for the measures that were validated (see Table 2 in Worksheet 2.3) and
recommendations for improving the medical record review process.
Step 5: Prepare for the MCP Onsite Visit

WORKSHEET 2.4

Before conducting onsite activities, the EQRO will
contact the MCP to:
•

Explain the procedures and timeline for
performance measure validation activities

Resources for Activity 1, Step 5
Worksheet 2.4. Potential Documents and
Process for Review
• Provides a checklist of documents, data,
and procedures the MCP should make
available before or during the onsite visit

•

Communicate the EQRO’s policies and
procedures for safeguarding confidential
information and signed confidentiality
agreements

•

Organize the onsite visit to ensure the availability of necessary documentation and staff
(See box, Potential Onsite Participants)

At this stage, the EQRO should also request confirmation of the list and description of staterequired performance measures. The EQRO will provide the MCP a list of documents, data, and
procedures that may be reviewed before or during onsite activities (refer to Worksheet 2.4.
Potential Documents and Processes for Review).
Potential Onsite Participants
During the onsite visit, the MCP should arrange for staff and vendors to meet with the EQRO to provide information
about the processes to processes to calculate or report performance measures. The EQRO may want to suggest to
the MCP that corporate staff—particularly Information Systems (IS) staff—be included in the onsite visit as corporate
staff may provide additional insight into some interview questions. Participants may include:
• The Director of Health/Medical Information Systems
• Information system programmers or operators
• Director of Member/Patient Services and staff
• Director of Utilization Management and staff
• Director of Quality Improvement and staff

PROTOCOL TWO | 67

ACTIVITY 2: CONDUCT ONSITE VISIT ACTIVITIES
Onsite visit activities provide an opportunity for the EQRO to follow up on findings from the preonsite information system assessment and to confirm or clarify information about the production
and reporting of performance measures through document review or direct observation (See
box, Purpose of the Onsite Visit).
During the onsite visit, the EQRO will complete the following steps:
1

Review the information systems underlying performance measurement

Assess data integration and control for performance measure calculation

Review performance measurement production

Complete the detailed review of measures

Assess the sampling process

Review preliminary findings and outstanding items
Purpose of the Onsite Visit
• Confirm, observe, and query systems used to produce performance measure results, including membership,
medical, pharmacy, provider, and other ancillary or supplemental data sources
• Investigate and follow up on issues identified from the ISCA
• Assess data integration and control procedures for accurate production of the performance measures
• Assess data completeness
• Confirm processes for calculating and reporting the performance measures

Step 1: Review Information Systems Underlying Performance Measurement

WORKSHEET 2.5

The review of the ISCA which had begun during the
pre-onsite phase continues onsite. During this phase,
the EQRO reviews the information system
components that the MCP uses to produce
performance measures via (1) staff interviews, (2)
primary source documents, (3) systems and
processes used to calculate performance measures,
(4) data entry observation, and (5) data files. These
sources are described below.

Resources for Activity 2, Step 1
Worksheet 2.5. Interview Guide for MCP
Data Integration and Control Personnel
• Provides a list of interview questions for
key staff involved in the production of
performance measures using questions
tailored to the MCP’s processes for
producing these measures. Tailor the
questions as appropriate

1. Staff Interviews
The EQRO will interview key staff (scheduled and confirmed ahead of the visit) involved in the
production of performance measures using questions tailored to the MCP’s processes for
producing performance measures based on findings from the ISCA. These interviews also
provide an opportunity to supplement the review of information system policies, procedures, and
data (described below). See Worksheet 2.5. Interview Guide for MCP Data Integration and
Control Personnel.
68 | PROTOCOL TWO

2. Primary Source Verification
The EQRO will review the primary source documents, including paper forms and other input to
the MCP systems, and confirm that the information from the primary source matches the
information used for performance measurement. In addition, the EQRO will review the
processes used to input, confirm entry, and identify errors, as well as processes used to
transmit and track the data through systems. Typical forms the EQRO will review include:
•

Member-initiated enrollment data

•

Hospital claims/encounters

•

Ambulatory claims/encounters

•

Prescription data

•

Practitioner demographic forms

•

Practitioner credentialing forms

•

Claims logs

•

Lab results

3. System and Process Review
The EQRO will review the MCP’s documentation describing the systems and processes used to
calculate performance measures to confirm they adhere to state policies and procedures. These
include systems and processes for collecting, storing, and reporting data. All documentation
received and examined must be recorded.
4. Observation
The EQRO will observe key MCP processes required for performance measure calculation to
assess data entry and other data manipulations. Examples include:
•

Data entry of membership updates, claims/encounter data, and practitioner data (e.g.,
confirm that mandatory fields are required and invalid data elements are identified, such as
invalid birth dates or invalid service dates)

•

Claims operations including overrides or exceptions

•

Computer operations and security plans to confirm procedures are followed

The EQRO will directly observe the Extract, Transform, and Load (ETL) process52 and its
replication by two separate operators through the process using an observation guide to
confirm the activities, as well as the process where data are incomplete (e.g., a claim without
a provider identification number).
5. Data File Review

ETL is when these three database functions (extract, transform, and load) are combined into one tool to pull data out of
one database and place it into another database.

PROTOCOL TWO | 69

The EQRO will directly examine data files to confirm the data are stored and processed
according to the documentation provided. Examples of files to review include:
•

Transaction files for clinical services, membership, and practitioner changes

•

Intermediate files created by extracts, queries, and analysis applications

•

Data repository files

Step 2: Assess Data Integration and Control for Performance Measure Calculation

WORKSHEET 2.6

In this step, the EQRO will assess the MCP’s ability to
link data from multiple sources and the extent to which
the MCP has created systems and processes to ensure
the accuracy of the calculated performance measures.
Worksheet 2.6. Data Integration and Control Findings
helps the EQRO review:

Resources for Activity 2, Step 2
Worksheet 2.6 Data Integration and
Control Findings Tool
• Guides the EQRO’s review of data
integration and control elements
during the onsite visit

•

Accuracy of data transfers to the assigned
performance measure repository

•

Accuracy of file consolidations, extracts, and derivations

•

Adequacy of the performance measure data repository to calculate and report performance
measures

•

Management of report production and reporting software

Step 3: Review Performance Measure Production

WORKSHEET 2.7
WORKSHEET 2.8

The EQRO will review the MCP’s documentation of all
steps undertaken in the production of the performance
measures, including:
•

•
•

Data collection from various sources (e.g.,
membership, enrollment, provider, claims, or
encounter files; medical records; laboratory,
pharmacy, or other ancillary records)
Steps taken to integrate the required data into a
performance measure data set or repository
Procedures or programs to query the data
set/repository to identify denominators, generate
appropriate samples, determine numerators, and
apply proper algorithms to the data in order to
produce valid and reliable performance measures

70 | PROTOCOL TWO

Resources for Activity 2, Step 3
Worksheet 2.7. Data and Processes Used
to Produce Performance Measures:
Documentation and Review Checklist
• Helps the EQRO check the
documentation of steps taken in the
production of the performance
measures
Worksheet 2.8. Data and Processes Used
to Produce Performance Measures:
Findings
• Template to record findings based on
measurement plans, policies, and
programming specifications

Step 4: Complete the Detailed Review of Measures

WORKSHEET 2.2
WORKSHEET 2.9
WORKSHEET 2.10
WORKSHEET 2.11

Resources for Activity 2, Step 4
Worksheet 2.2. Performance Measure Validation Template
• Template for documenting audit specifications for the validation components of each performance measure listed in
Worksheet 2.1., and to assess the MCP’s measurement and reporting process for each component
Worksheet 2.9. Policies, Data, and Information Used to Produce Measures: Review Checklist
• Checklist that tracks documents and data used to assess the accuracy of the MCP’s performance measure
calculations
Worksheet 2.10. Measure Validation Findings
• Documents adherence to guidance for the denominator; programming logic, source code, and calculations;
identifying medical events; exclusion criteria; population estimates; identifying the at-risk population; inclusion of
qualifying events in the numerators; and medical record data in the numerator
Worksheet 2.11. Interview Guide for Assessing Processes and Procedures Used to Produce Numerators and
Denominators
• Provides a list of interview questions that can be tailored to supplement findings recorded in Worksheet 2.10
For performance measures requiring medical record review, please use Worksheet 2.3. Medical Record Review
Validation Tool. The EQRO should validate the results of the medical record review for 30 enrollees who met the
numerator requirements for at least two measures. For more information, refer to Activity 1, Step 4.

In Step 4, the EQRO determines the extent to which the MCP correctly used the technical
specifications to produce accurate performance measure results. All validation components
should be addressed during this step using Worksheet 2.2. Performance Measure Validation
Template (or similar tool).
To ensure the integrity and comparability of the performance measures, the EQRO should pay
special attention to factors affecting the accuracy and completeness of the denominators and
numerators. For example, the EQRO should assess whether the MCP used the appropriate
data and methods to identify the entire eligible population for the denominator (including linkage
of data from separate sources, application of inclusions and exclusions, and creation of complex
episodes, where applicable). In addition, the EQRO should determine whether the MCP
correctly identified and assessed qualifying medical events for the numerator to include all
appropriate events, while excluding events that do not qualify. The EQRO should determine
whether the numerators and denominators were calculated appropriately based on all
applicable codes (such as diagnoses, procedures, and prescription drugs) and all available data
sources (such as membership/enrollment data, claim/encounter data, provider data, utilization
or medical management information systems data, or data extracted from medical records).

PROTOCOL TWO | 71

Step 5: Assess the Sampling Process (if applicable)

WORKSHEET 2.12
WORKSHEET 2.13

Resources for Activity 2, Step 5
Worksheet 2.12. Policies, Procedures, and Data Used to Implement Sampling: Review Checklist
• This Review Checklist guides this review by providing a list of documents, data, and procedures to assess the
sampling process
Worksheet 2.13. Sampling Validation Findings
• For each measure involving a sample, this worksheet helps assess the extent to which:
○ The MCP followed the specified sampling method to produce an unbiased sample representative of the entire
included population
○ The MCP maintains its performance measurement population sample frame to allow for a sample to be re-drawn
or used as a source for replacement
○ Sample sizes collected conform to the methodology in the performance measure specifications
○ The sample is representative of the entire population
○ Proper substitution methodology is followed for performance measures that include medical record reviews

The EQRO will determine whether the sample represents the entire eligible population in all
relevant dimensions. The MCP’s sampling method should not exclude any population
subgroups to which the performance measure applies. For example, when assessing well-child
care, the sample should not exclude children with special health care needs whose primary care
provider is a specialist other than a pediatrician or family practitioner.
Step 6: Communicate Preliminary Findings and Outstanding Items

At the conclusion of the onsite visit, the EQRO will communicate preliminary findings to the
MCP, including any outstanding items for follow-up. The information communicated during the
closing conference will appear in the EQRO’s subsequent preliminary report to the MCP. In
addition, the EQRO should provide a list of outstanding items before completing the preliminary
report to allow the MCP the maximum time to resolve identified issues.

ACTIVITY 3: CONDUCT POST-ONSITE VISIT ACTIVITIES
WORKSHEET 2.3
WORKSHEET 2.6
WORKSHEET 2.10
WORKSHEET 2.13

72 | PROTOCOL TWO

Post-onsite visit activities focus on assessing
MCP corrective actions and reporting findings to
the state using the format and timeframes
established by the state. The EQRO will:
1

Analyze all data and submit a preliminary
report to the MCP detailing areas of
concern, suggested methods for correction,
and a timeline for the MCP to make
corrections
Re-validate selected performance measures
and the measurement processes the MCP
used to make corrections
Re-evaluate the corrected information and
submit a report of validation findings to the
state

Determine preliminary validation findings for
each measure

Assess accuracy of MCP’s performance
measure reports to the state, and

Submit the validation report to state

Note that throughout this EQR-related activity or
during any part of an EQR, the state may decide
that immediate corrective action is required.

Resources and Tools for Activity 3
Information gathered in Activities 1 and 2 using the
following worksheets and tools may be helpful when
preparing the final validation report:
Worksheet 2.3. Medical Record Review Validation
Template
• Describes procedures and sample tools for
validating medical review findings
Worksheet 2.6. Data Integration and Control
Findings Tool
• Provides a template for recording findings from
interviews with data integration and control MCP
personnel
Worksheet 2.10. Measure Validation Findings
• Provides a template for recording findings from
the measures record validation review
Worksheet 2.13. Sampling Validation Findings
• Provides a template to record findings from the
sampling assessment process
Worksheet 2.14. Framework for Summarizing
Information About Performance Measures
• Provides a template for summarizing information
about performance measure results

Step 1: Determine Preliminary Validation Findings for Each Measure

In the preliminary validation findings report, the EQRO will document findings, identify areas of
concern, and make suggestions for corrective action or long-term improvement for each of the
performance measures the EQRO validated. The report should indicate which MCP
performance measures and elements of the measures were invalid and therefore, should not be
reported (if any). The report should also provide the MCP with correctional guidance for
improving the overall measure production process. In addition to communicating written
findings, the EQRO may participate in meetings with key MCP personnel responsible for the
calculation and reporting of performance measures to assist the MCP with implementing
recommended corrective action.
Once the EQRO has submitted its preliminary findings to the MCP, the MCP may offer
comments and documentation to correct errors and omissions in the EQRO’s preliminary report.
At the discretion of the state, the MCP may recalculate performance measures based on the
findings. The EQRO must then revalidate the revised performance measure(s) and incorporate
the MCP’s comments or revised performance measure validation findings. If the state chooses
not to allow measure re-validation, the recommendations will be reviewed in the following year
as part of the MCP assessment of progress toward recommended improvements.

PROTOCOL TWO | 73

Step 2: Assess and Document the Accuracy of Performance Measure Reports

The EQRO will assess and document the extent to which the MCP reported the calculated
performance measures correctly in its final report to the state, and verify the reporting of each
performance measure by reviewing:
•

Procedures for submitting reports that meet state requirements (such as specified format,
supporting documentation, and timing)

•

Documentation that the MCP appropriately implemented procedures to properly submit
required reports to state

Step 3: Submit Validation Report to the State

WORKSHEET 2.14

The EQRO will always use the state’s decision rules for determining the degree to which each
of the MCP’s reported performance measures are accurate and complete. The decision rules for
compliance should be consistent across MCPs within the state. The final report should follow
the state’s required format, and include the following elements:
•

A list of the measures validated by the EQRO

•

A description of the EQRO’s validation activities including:
○

The EQRO team members involved in the validation

○

A summary of the validation strategy

○

The data collection methods and analysis

○

List of onsite participants (EQRO, MCP, and vendor)

○

Other considerations relevant to the onsite visit process

•

Worksheets, tools, and other supporting documentation

•

Analyses and conclusions based on the validation process for each performance measure
including:

•

○

The validation status of each performance measure (including the results of the
medical record review)

○

Actual results of the performance measures (not just the results of the validation)

○

Findings on the MCP’s information systems capabilities and data integration, including
documentation of the timing of the state’s most recent ISCA and a description of what
documentation was reviewed by the EQRO

Recommendations for improving the process for calculating and reporting performance
measures, including implications for the MCP’s data systems, methods, and staffing (e.g.,
programming and analytic capacity)

When possible, the validation report should also identify recommendations from the previous
year’s report submitted to the state, and discuss progress made on these recommendations
over the past year based on information gathered during the validation process.
END OF PROTOCOL 2
74 | PROTOCOL TWO

WORKSHEETS FOR PROTOCOL 2:
PERFORMANCE MEASURE VALIDATION TOOLS
Instructions. Use these or similar worksheets to assist in validating performance measures reported by the MCP.
These worksheets identify the performance measure to be validated, provide templates for validating performance
measures, and provide tools for conducting pre-onsite, onsite, and post-onsite visit activities. This tool includes the
following worksheets crosswalked to the applicable Activity and Step:
Worksheet name

Protocol activity and step

Worksheet 2.1. List of Performance Measures to be
Validated

Activity 1. Step 1. Define the Scope of the Validation

Worksheet 2.2. Performance Measure Validation
Template

Activity 1. Step. 1. Define the Scope of the Validation
Activity 1. Step 3. Conduct Detailed Review of
Measures
Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.3. Medical Review Validation Template

Activity 1. Step 4. Initiate Review of Medical Record
Data Collection
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.4. Potential Documents and Process for
Review

Activity 1. Step 5. Prepare for the MCP Onsite Visit

Worksheet 2.5. Interview Guide for MCP Data
Integration and Control Personnel

Activity 2. Step 1. Review Information Systems
Underlying Performance Measurement

Worksheet 2.6. Data Integration and Control Findings
Tool

Activity 2. Step 2. Assess Data Integration and Control
for Performance Measure Calculation
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.7. Data Processes Used to Produce
Performance Measures: Documentation and Review
Checklist

Activity 2. Step 3. Revie Performance Measure
Production

Worksheet 2.8. Data and Processes Used to Produce
Performance Measures: Findings

Activity 2. Step 3. Review Performance Measure
Production

Worksheet 2.9. Polices, Data, and Information Used to
Produce Measures: Checklist

Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.10. Measure Validation Findings

Activity 2. Step 4. Complete the Detailed Review of
Measures
Activity 3. Conduct Post-Site Visit Activities

Worksheet 2.11. Interview Guide for Assessing
Processes Used to Produce Numerators and
Denominators

Activity 2. Step 4. Complete the Detailed Review of
Measures

Worksheet 2.12. Policies, Procedures, and Data Used
to Implement Sampling: Review Checklist

Activity 2. Step 5. Assess the Sampling Process (if
applicable)

Worksheet 2.13. Sampling Validation Findings

Activity 2. Step 5. Assess the Sampling Process (if
applicable)
Activity 3. Conduct Post-Onsite Visit Activities

Worksheet 2.14. Framework for Summarizing
Information about Performance Measures

Activity 3. Step 3. Conduct Post-Onsite Visit Activities

PROTOCOL TWO | 75

For each MCP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
EQRO interview date
Type of delivery system (check all that apply)

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM entity
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

Note: IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care Organization;
PIHP = Prepaid Inpatient Health Plan; PCCM = Primary Case Management.

76 | PROTOCOL TWO

Worksheet 2.1. List of Performance Measures to be Validated
Instructions. This worksheet is used to identify the measures to be validated, the data source, reporting frequency,
and format as described in Activity 1. Step 1. Complete the worksheet for each measure to be validated, and adapt
as needed. The list below is illustrative of the performance measures that could be included in the worksheet.

Performance
Measures
(Illustrative)

NQF #

Childhood
Immunization
Status (CIS-CH)
(NCQA):
Combo 3

0038

Immunizations
for Adolescents
(IMA-CH)
(NCQA):
Combo 1

1407

Well-Child Visits
in the First 15
Months of Life
(W15-CH)
(NCQA)

1392

Well-Child Visits
in the Third,
Fourth, Fifth,
and Sixth Years
of Life (W34CH) (NCQA)

1516

Admin.
Data

Medical
Record
Review
(MRR)

Hybrid
(Admin.
Data and
MRR)

Electronic
Health
Record

Survey

Reporting
Frequency
and
Format

For
example,
through
HEDIS®
Interactive
Data
Submission
System
(IDSS) by
February
2018

PROTOCOL TWO | 77

Comments

Worksheet 2.2. Performance Measure Validation Template
Instructions. For each performance measure, use this template to gather audit specifications for the validation
components (as described in Activity 1. Steps 1 and 3, and Activity 2. Step 4) and to assess the MCP’s measurement
and reporting process for each component.
For each validation component, indicate whether the measure meets validation requirements by checking “Yes,” “No,”
or “Not applicable.” Insert comments to explain “Not met” and “Not applicable” responses. Use the following guidance
to assess each component.
•

Yes: The MCP’s measurement and reporting process was fully compliant with state specifications

•

No: The MCP’s measurement and reporting process was not fully compliant with state specifications. This
designation should be used for any validation component that deviates from the state specifications, regardless of
the impact of the deviation on the final rate. All components with this designation must include an explanation of
the deviation in the comments section

•

Not applicable: The validation component was not applicable. Include an explanation in the comments section
(e.g., sampling not required, medical record review not included)

Managed Care Plan __________________________________________________________________
Performance Measure __________________________________________________________
Method for Calculating Measure: [ ] Admin

Validation component
Documentation:
Did appropriate and complete
measurement plans and
programming specifications
exist, including data sources,
programming logic, and
computer source code?
Were internally developed
codes used?
Denominator:
Were all the data sources used
to calculate the denominator
complete and accurate (e.g.,
eligibility files, claims
files/encounter data, medical
records, provider files,
pharmacy records, including
those for members who
received services outside the
MCP’s network)?

78 | PROTOCOL TWO

[ ] Medical Record Review [ ] Hybrid [ ] EHR [ ] Survey

Audit specifications

Yes

Not
applicable

Comments

Validation component

Audit specifications

Yes

Not
applicable

Comments

Did the calculation of the
performance measure adhere to
the specifications for all
components of the denominator
(e.g., member ID, age, sex,
continuous enrollment, clinical
codes such as ICD-10, CPT® 53,
DRGs, member
months/member years, and
adherence to the measurement
period)?
Numerator:
Were the data sources used to
calculate the numerator
complete and accurate (e.g.,
claims files, medical records,
provider files, pharmacy
records, including those for
members who received services
outside the MCP’s network)?
Did the calculation of the
performance measure adhere to
the specifications for all
components of the numerator
(e.g., member ID, clinical codes
such as ICD-10, CPT®, LOINC,
DRGs, pharmacy data, relevant
time parameters such as
admission/discharge dates or
treatment start and stop dates,
adherence to the measurement
period, number or type of
provider)?
If medical record abstraction
was used, were the abstraction
tools adequate?
If the hybrid method was used,
was the integration of
administrative and medical
record data adequate?
If the hybrid method or medical
record review was used, did the
results of the medical record
review validation substantiate
the reported numerator?
Sampling:
Was the sample unbiased?
Did the sample treat all
measures independently?
Did the sample size and
replacement methodologies
meet specifications?

PROTOCOL TWO | 79

Validation component

Audit specifications

Yes

Not
applicable

Comments

Reporting:
Were the state specifications for
reporting performance
measures followed?
Overall assessment: In the
comments section, note any
recommendations for the MCP’s
measurement and reporting
process

Yes
(Please
explain)

Additional Audit Questions
Were any members excluded for contraindications found in the administrative data?
Were any members excluded for contraindications found during the medical record
review?
Were internally developed codes used?

What is the estimated impact of data incompleteness on
the rate(s) calculated for this measure?
• 0–5 percentage points
• >5–10 percentage points
• >10–20 percentage points
• >20–40 percentage points
• >40 percentage points
• Unable to determine
What is the direction of the bias? (Check one)
• Over-reporting
• Under-reporting
• Not applicable (no bias detected)
What documentation was used to estimate the above
percentage (e.g., internal reports, studies, comparison to
medical records)?

80 | PROTOCOL TWO

Check
one

Comments

Overall Validation Finding
Provide an overall validation finding for each performance measure. The validation finding is determined by the
magnitude of the errors detected for the audit elements, not by the number of audit elements determined as “NO.”
Consequently, it is possible that an error for a single audit element may result in a designation of “Do Not Report”
(DNR) because the impact of the error materially biased the reported performance measure. Conversely, it is also
possible that several audit element errors may have little impact on the reported rate and, thus the measure is
“Reportable” (R).
Performance Measure Validation Finding (check one)

Comments

[ ] R = Reportable; measure was compliant with state
specifications
[ ] DNR = Do not report; MCP rate was materially biased and
should not be reported
[ ] NA = Not applicable; the MCP was not required to report the
measure
[ ] NR = Measure was not reported because the MCP did not
offer the required benefit
Overall assessment: In the comments section, note justification
and recommendations for the validation finding.

PROTOCOL TWO | 81

Example of Worksheet 2.2. Performance Measure Validation Template
Below is an example of a completed, customized performance measure validation worksheet similar to what an
EQRO would prepare before its onsite visit. This worksheet is based on the Child and Adult Core Set specifications
for the performance measure.
Performance Measure Chlamydia Screening in Women Ages 21 – 24 (CHL-AD) (NCQA)__________________
Method for Calculating Measure: [ X ] Admin

Validation component

[ ] Medical Record Review [ ] Hybrid [ ] EHR [ ] Survey

Audit specifications

Yes

• Obtain and review all file
layouts, code,
documentation

Documentation:
Did appropriate and complete
measurement plans and
programming specifications
exist, including data sources,
programming logic, and
computer source code?

• Code and
documentation mapped
to measure specification

Denominator:
Were all the data sources used
to calculate the denominator
complete and accurate (e.g.,
eligibility files, claims files,
provider files, pharmacy
records)?
Did the calculation of the
performance measure adhere to
the specifications for all
components of the denominator
(e.g., member ID, age, sex,
continuous enrollment, clinical
codes such as ICD-10, CPT®,
DRGs, member
months/member years, and
adherence to the measurement
period)?

• Medicaid population
appropriate segregated
from
commercial/Medicare
• Population defined as
active Medicaid
enrollment as of 12/31
of measurement year
• Members ages 16-24 as
of 12/31 of the
measurement year
• Only females selected
• Members enrolled in
MCP on 12/31 of the
measurement year.
• Continuously enrolled
from 1/1 to 12/31 of the
measurement year with
no more than one break
of up to 45 days
allowed.
• Shifts between Medicaid
and CHIP enrollment
were not counted as
breaks; shifts between
Medicaid/CHIP and
commercial enrollment
were counted as breaks.

82 | PROTOCOL TWO

Not
applicable

Comments

Validation component

Audit specifications

Yes

Were the data sources used to
calculate the numerator
complete and accurate (e.g.,
claims files, medical records,
provider files, pharmacy
records, including those for
members who received services
outside the MCP’s network)?

• Sexually active based
on pharmacy and
claims/encounter data

Did the calculation of the
performance measure adhere to
the specifications for all
components of the numerator
(e.g., member ID, clinical codes
such as ICD-10, CPT®, LOINC,
DRGs, pharmacy data, relevant
time parameters such as
admission/discharge dates or
treatment start and stop dates,
adherence to the measurement
period, number or type of
provider)?

• Exclusions were
performed according to
state specifications

Not
applicable

Comments

Numerator:

• Properly identified
enrollees. Based on the
ISCA findings, the data
sources used for the
numerator were
accurate
X

• Only the codes listed in
specifications as defined
by state were counted
as exclusions
• Standard codes listed in
state specifications
(and/or properly mapped
internally developed
codes) were used
• Members were counted
only once; double
counting was prevented
• Service performed
between 1/1 and 12/31
of the measurement
year

If medical record abstraction
was used, were the abstraction
tools adequate?

Not applicable; no
medical record
abstraction

If the hybrid method was used,
was the integration of
administrative and medical
record data adequate?

Not applicable;
hybrid not used

If the hybrid method or medical
record review was used, did the
results of the medical record
review validation substantiate
the reported numerator?

Not applicable;
hybrid and MRR not
used

Did the sample treat all
measures independently?

Not applicable; no
sampling

Did the sample size and
replacement methodologies
meet specifications?

Not applicable; no
sampling

Sampling:

PROTOCOL TWO | 83

Validation component

Audit specifications

Yes

• Measure-eligible
population is accurate
and documented
(inclusions, exclusions)

Not
applicable

Comments

Reporting:
Were the state specifications for
reporting performance
measures followed?

• Method is accurate and
documented
(measurement period,
data source)
• Information on
numerator, denominator,
rate is accurate and
documented
• Deviations (if any) are
accurate and
documented
Overall assessment: In the
comments section, note any
recommendations for the MCP’s
measurement and reporting
process

Additional Audit Questions

Measure meets all
audit specifications
and is reportable by
the state

Yes
(Please
explain)

Were any members excluded for contraindications found in the administrative data?

Were any members excluded for contraindications found during the medical record
review?

Were internally developed codes used?

What is the estimated impact of data incompleteness on the rate(s) calculated for this
measure? (Check one)
• 0–5 percentage points

• >5–10 percentage points
• >10–20 percentage points
• >20–40 percentage points
• >40 percentage points
• Unable to determine
What is the direction of the bias? (Check one)
• Over-reporting
• Under-reporting
• Not applicable (no bias detected)

What documentation was used to estimate the above percentage (e.g., internal
reports, studies, comparison to medical records)?

Internal reports

84 | PROTOCOL TWO

Performance Measure Validation Finding (check one)

Comments

PROTOCOL TWO | 85

Worksheet 2.3. Medical Record Review Validation Template
Instructions. This template provides instructions for conducting the medical record review (as described in Activity 1.
Step 4 and Activity 3) and two tables to summarize re-abstraction findings from the review (Table 1) and record the
impact of findings from the review (Table 2).
The purpose of medical record review (MRR) validation is to verify the accuracy of the MRR conducted by each MCP.
For each of at least two measures that included medical record review, the EQRO will validate the medical records of
30 enrollees found to meet numerator requirements. In states with separate Medicaid and CHIP programs, the EQRO
will review 30 enrollees in each CHIP MCP and 30 enrollees in each Medicaid MCP for each of at least two measures
that included medical record review. Only those members included in a hybrid sample will be selected—the EQRO
will not conduct medical record audits to validate administrative data.
For each measure in which medical record review was used, the EQRO will request a list of all of the members in the
MCP’s MRR sample. From that list:
•

The EQRO will identify a sample of 30 members who meet numerator requirements

•

MCPs will then be asked to provide access to or copies of medical records so that the EQRO can verify that each
member was appropriately included in the denominator and received the required numerator service(s)

•

In cases where there are fewer than 30 numerator positives, the EQRO will review all records for that measure

To provide sufficient time for each MCP to gather the required medical record documentation, the EQRO may direct
the MCPs to submit their lists of members in their hybrid sample twice— the first list as a preliminary submission and
the second list as a final submission:
•

Submitting a first list before completion of the MRR process would allow an MCP additional time to retrieve
medical record documentation

•

Soon after receipt of the first list, the EQRO will provide the MCP with the list of medical records for which
documentation must be submitted

•

Only a portion of the 30 medical records for the validation sample will be included in the EQRO’s first sample
request list

•

The remainder of the 30 records will be selected from the final list. While the first submission of MRR findings is
optional, it is recommended

The EQRO should accept the first list submission approximately one month before the scheduled audit or another
time specified by the EQRO. If an MCP chooses to submit a first list of medical records, it must still submit a final
listing sufficiently in advance of the scheduled audit as directed by the EQRO. For each submission:
•

MCPs will need to identify all members for whom MRR has been conducted and indicate which members have
been found to be numerator positives through MRR

•

The final list must reflect the MCP’s final medical record review findings, with members for whom a medical record
was never found identified as not having met the numerator requirements

No predetermined “passing” grade is set for the medical record audit. Rather, onsite auditors will use the MRR results
to determine if the hybrid rate (or solely MRR rate if applicable) is biased, and to what extent that bias affects the final
reported rate for that measure. The EQRO will identify to the state what effects bias, as well as incomplete data, will
have on the MCP’s calculation of the performance measure. For each of the evaluated measures, auditors will
determine the impact of the findings from the MRR validation process on the MCP’s Final Audit Designation.

86 | PROTOCOL TWO

Step 1: Calculate the Medical Record Review Error Rate
The EQRO will review up to 30 records identified by the MCP as meeting numerator requirements (as determined
through MRR) for the measures audited. Records are randomly selected from the entire population of MRR
numerator positives identified by the plan, as indicated on the MRR numerator listings submitted to the EQRO:
•

If fewer than 30 medical records are found to meet numerator requirements, all records are reviewed

•

Administrative numerator positives are not included as part of this validation process

The EQRO will calculate a MRR error rate for each performance measure calculated by the hybrid method or solely
from MRR as illustrated in Table 1.
Table 1: Summary of Medical Record Review (MRR) Re-abstraction Findings
Column A

Column B

Column C

Column D

Performance
measure

Number of MMR
positives
selected for
audit

Number of
medical records
received

Number of
medical records
found compliant

Column E

Column F

Accuracy rate
(%)
(col. D/col. B)

Error rate (%)
(100% - col. E)

Column A: Name of performance measure
Column B: Total number of MRR numerator positive records re-abstracted by EQRO as part of the medical record
review validation process (i.e., 30, or the total population if less than 30 MRR numerator positives were
reported)
Column C: Total number of medical records submitted to EQRO as part of the medical record review validation
process (i.e., should be equal to Column B or less than Column B if one or more records were not
submitted on time)
Column D: Total number of medical records reviewed by EQRO and identified as meeting numerator requirements
Column E: Accuracy rate = percent of records selected for audit that were identified as meeting numerator
requirements (Column D/Column B)
Column F: Error rate = percent of records selected for audit that were identified as not meeting numerator
requirements (100% - Column E)

Step 2: Determining the Potential Impact of Medical Record Review Re-abstraction Findings on
Final Audit Designations
The next step in MRR validation is to determine whether any medical record review errors significantly biased the
final reported rate for a given performance measure. To make this determination, the EQRO, as directed by the state,
should develop and follow decision rules such as the following:
Sample Decision Rules:
•

Error Rate of 10 Percent or Less. If the error rate (Table 1, column F) is 10 percent or less, then the measure
automatically passes the MRR validation. The Final Audit Designation is then determined based on the auditors’
findings from the ISCA conducted as Pre-Onsite Visit Activity 3 and Onsite Visit Activity 1. As long as no errors
leading to significant bias are discovered during the other components of the audit process, the final rate is
considered as having met the validation standards

PROTOCOL TWO | 87

•

Error Rate of Greater than 10 Percent. If the error rate (Table 1, column F) is greater than 10 percent, then the
auditors determine the impact of the MRR validation findings on the final reported rate for the measure. For each
of the measures under review, auditors evaluate the impact of the MCP’s MRR processes on its final reported
rate by extrapolating findings from the audited medical record sample to the universe of all MRR positives. Details
on this process are in Table 2

•

The maximum amount of bias allowed for the final rate to be considered reportable is “X” percentage
points (to be determined by each state). If the amount of error in the MCP’s MRR process (Table 2, line 8)
does not cause the final reported rate to be biased by more than X percentage points, then the measure passes
the MRR validation. The compliance designation is then determined based solely on the auditors’ findings from
the ISCA. As long as no errors leading to significant bias are discovered during the other components of the
performance measure audit process, the final rate is considered valid

•

If the amount of error in the MCP’s medical review process (Table 2, line 8) ultimately causes the final
reported rate to be biased by more than X percentage points, the rate is automatically considered invalid. The
performance measure is then designated as invalid

Table 2: Impact of MRR Findings
Line #

Description

Final data collection method used (e.g., MRR,
hybrid)

Error rate (percentage of records selected for audit
that were identified as not meeting numerator
requirements, as shown in Table 1, column F)

Is error rate <10%? (Yes or No)

Measure A

Measure B

Measure C

• If yes, MCP passes MRR validation; no further
MRR calculations necessary
• If no, the full table must be completed to
determine the impact on the final rate
4

Denominator (the total number of members
identified for the denominator of this measure, as
identified by the MCP)

Weight of each medical record (impact of each
medical record on the final overall rate; determined
by dividing 100% by the denominator in line 4)

Total number of MRR numerator positives
identified by the MCP using MRR

Expected number of false positives
(Estimated number of medical records
inappropriately counted as numerator positives;
determined by multiplying the error rate in line 2 by
line 6, the total number of MRR numerator
positives reported)

Estimated bias in final rate
(The amount of bias caused by medical record
review, measured in percentage points;
determined by multiplying the expected number of
false positives in line 7 by line 5, the weight of each
medical record)

•

If line 8 is <=X%, then the final rate is not considered to be significantly biased by MRR alone. If the other
components of the audit process did not identify any other issues that would introduce bias into the rate, the rate
will be considered valid

•

If line 8 is >X%, then the final rate is considered to be significantly biased. The measure will be considered invalid

88 | PROTOCOL TWO

Worksheet 2.4. Potential Documents and Processes for Review
Instructions. To assist the EQRO in assessing the MCP’s information system and validity of reported performance
measures, this worksheet provides a checklist of documents, data, and procedures the MCP should make available
before or during the onsite visit (as described in Activity 1, Step 5). Record any questions or concerns raised by the
review of documents and/or processes, and any specific checks or tests the EQRO would like to conduct or have
demonstrated during the onsite visit. The EQRO can use its discretion in selecting which ones to review.
For example:
•

Compare samples of data in the repository to transaction files. Are any members, providers, or services lost in the
process?

•

Is the required level of coding detail maintained (e.g., all significant digits, primary and secondary diagnoses
remain)?

•

If the MCP uses a performance measure repository, review the repository structure. Does it contain all the key
information necessary for performance measure reporting?

•

How does the MCP test the process used to create the performance measure reports?

•

Does the MCP use any algorithms to check the reasonableness of data integrated to report the MCP-level
performance measures?

•

Examine report production logs and run controls. Is there adequate documentation of the performance measure
report generation process? How are report generation programs documented? Is there version control in place?

Checklist of documents and processes for review

Reviewed?
Y/N

Comments for onsite visit

1. Data integration and control
Procedures and standards for all aspects of the data
repositories used in producing performance measures,
including building, maintaining, managing, and testing
performance measures
Manuals that include application system development
methodology, database development, and design and decision
support system utilization
System documentation including flow charts and codes for
backups, recovery, archiving, and other control functions
Procedures to consolidate information from disparate
transaction files
Record and file formats and descriptions, for entry,
intermediate, and repository files
Electronic formats and protocols
Electronic transmission procedures documentation
Processes to extract information from the repositories
Source code data entry, data transfer, and data manipulation
programs and processes
Descriptive documentation for data entry, transfer, and
manipulation programs and processes
If applicable, procedures for coordinating vendor activities to
safeguard the integrity of the performance measurement data

PROTOCOL TWO | 89

Checklist of documents and processes for review
Samples of data from repository and transaction files to assess
accuracy and completeness of the transfer process
Comparison of actual results from file consolidation and data
abstracts to those which should have resulted according to
documented algorithms
Documentation of data flow among vendors to assess the
extent to which there was proper implementation of procedures
to safeguard the integrity of the performance measure data
Documentation of data cut-off dates
Documentation of proper run controls and of staff review of
report runs
Copies of files and databases used for performance measure
calculation and reporting
Procedures governing production process for MCP
performance measures, including standards and schedules
2. Collection, calculation, and documentation of
performance measures
Policies for the documentation of data requirements, data
issues, validation efforts, and results
A project or measurement plan for each performance measure
Documentation of programming specifications, including work
flow, data sources, and uses which include diagram or
narrative descriptions
Documentation of the original universe of data that includes
record-level patient identifiers, which can be used to validate
programming logic for creating denominators, numerators, and
samples
Documentation of computer queries, programming logic, or
source code used to create final denominators, numerators,
and interim data files
Documentation that includes dated job log or computer run for
denominators and numerators, with record counts for each
programming step and iteration
Documentation of medical record review including:
• Qualifications of medical record review supervisor and staff
• Reviewer training materials
• The use of audit tools, including completed copies of each
record-level reviewer determination
• All case-level critical performance measure data elements
used to determine a positive or negative event or exclude a
case, and
• Interrater reliability testing procedures and results
Documentation of statistical test results and any corrections or
adjustments to data along with justification for such changes
Documentation of sources of any supporting external data or
prior years’ data used in reporting

90 | PROTOCOL TWO

Reviewed?
Y/N

Comments for onsite visit

Checklist of documents and processes for review

Reviewed?
Y/N

Comments for onsite visit

Policies to assign unique membership ID that allows all
services to be properly related to the specific appropriate
enrollee, despite changes in status, periods of enrollment or
disenrollment, or changes across product lines (e.g., CHIP and
Medicaid).
Procedures to identify, track, and link member enrollment by
product line, product, geographic area, age, sex, member
months, and member years
Procedures to track individual members through enrollment,
disenrollment, and possible re-enrollment
Procedures to track members through changes in family
status, changes in benefits or managed care type (if they
switch between Medicaid coverage and another product within
the same MCP)
Methods to define start and cessation of coverage
Procedures to link member months to member age
Description of software or programming languages used to
query each database
Description of software used to execute sampling of population
files when sampling is used
Member database
Provider data (including facilities, labs, pharmacies,
physicians, etc.)
Database record layout and data dictionary
Survey data used for performance measures (See Protocol 6)
Policies to maintain files from which the samples are drawn in
order to keep population intact in the event that a sample must
be re-drawn, or replacements made
Computer source code or logic identifying specified sampling
techniques and documentation that the logic matches the
specifications set forth for each performance measure,
including sample size and exclusion methodology
Methods used for sampling for measures calling for medical
record or hybrid data
Documentation assuring that sampling methodology treats all
measures independently and that there is no correlation
between drawn samples
Observation or documentation of procedures in which a biased
sample was identified and corrected
Documentation of “frozen” or archived files from which the
samples were drawn, and if applicable, documentation of the
MCP’s process to re-draw a sample or obtain necessary
replacements
For performance measures that are easily under-reported,
procedures to capture data that may reside outside the MCP’s
datasets
Procedures for mapping non-standard codes to standard
coding

PROTOCOL TWO | 91

Checklist of documents and processes for review
Policies, procedures, and materials that provide evidence of
proper training, supervision, and adequate tools for medical
record abstraction tasks (this may include medical record
abstraction tools, training material, checks of inter-rater
reliability, etc.)
Procedures for assuring that combinations of record-review
data with administratively determined data are consistent and
verifiable
Evidence that MCP’s use of codes to identify medical events
were correctly evaluated when classifying members for
inclusion or exclusion in the numerator
Evidence that MCP has counted each member and/or event
only once
Programming logic or demonstration that confirms that any
non-standard codes used in determining the numerator have
been mapped to a standard coding scheme in a manner that is
consistent, complete, and reproducible
Programming logic or source code that identifies the process
for integrating administrative and medical record data for
numerator
Procedures for properly executing complex medical algorithms,
such as
• Claim-dependent events
• Events that require matching claims and pharmacy data
• Events that require matching visit codes, and
• Events that require accurately identifying and computing
multiple numerator events
Procedures for displaying denominator counts, numerator
counts, precision levels, sums and cross-totals
Procedures for reporting small sample sizes (to be consistent
with required methodology established by state)
Programming logic and/or source code for arithmetic
calculation of each measure
Review of reported measures to assess consistency of
common elements (e.g., membership counts, number of
pregnancies and births, etc.)
Programming logic and/or source code for measures with
complex algorithms, to ensure adequate matching and linkage
among different types of data
Documentation showing confidence intervals of calculations
when sampling methodology used
Documentation showing calculation of levels of significance of
changes
Procedures for submitting reports that meet state requirements
(e.g., specified electronic format, supporting documentation,
and timing)
Documentation that procedures for properly submitting
required reports to state were implemented appropriately

92 | PROTOCOL TWO

Reviewed?
Y/N

Comments for onsite visit

Worksheet 2.5. Interview Guide for MCP Data Integration and Control Personnel
Instructions. As part of the EQRO’s review of information system components the MCP uses to produce
performance measures, the EQRO should interview key staff (including appropriate vendor staff) involved in the
production of performance measures using questions tailored to the MCP’s processes for producing those measures.
These interviews are an opportunity to supplement the review of information system policies, procedures, and data
(as described in Activity 2, Step 1). Please tailor the questions below as appropriate.
MCP Contact and Background Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, email address, and date of interview, if applicable.
MCP name:
MCP contact name(s):
Title(s):
Mailing address:
Phone number(s):
Email address:
Interview Date:
EQRO reviewers:
Year of first Medicaid enrollment:
Year of first CHIP enrollment:
Year of first MCP performance report
(any product line)
1. Has the MCP previously undergone validation of its state performance measure reporting process? If so, when
did the validation take place and who conducted it?
2. How is performance measure data collection accomplished? (Check all that apply)
[

] By querying the applicable information system on-line

[ ] By using extract files created for analytical purposes? If so, how frequently are the files updated? How
do they account for claim/ encounter for accuracy?
[ ] By using a separate relational database or data warehouse? If so, is this the same system from which
all other reporting is produced?
[ ] Reports created from an NCQA-certified vendor software product? If so, how frequently are the files
updated? How are reports checked for accuracy?
3. Review the procedure(s) for consolidating claims/encounter, member, provider, and other data necessary for
performance reporting (whether it be into a relational database or file extracts on a measure-by-measure basis):
• How many different sources of data are merged together to create reports?
• What control processes are in place to ensure that this merger is accurate and complete?
4. How does the MCP test the process used to create the performance measure reports?

PROTOCOL TWO | 93

5. Does the MCP use any algorithms to check the reasonableness of data integrated to report the MCP performance
measures?
6. Is performance measurement reporting programming reviewed by supervisory staff?
7. Please describe any internal backup for performance measure programmers, if one exists. Do others know the
programming language and the structure of the actual programs? Please describe what documentation exists, if
any.
8. How does the MCP prevent loss of claim and encounter data when systems fail?
9. Please describe the administrative data backup systems are in place.
10. What types of authorization are required to be able to access claims/encounter, provider, membership, and
performance measure repository data?
11. Please describe documentation review and demonstrations provided.

94 | PROTOCOL TWO

Worksheet 2.6. Data Integration and Control Findings Tool
Instructions. During the onsite visit (described in Activity 2, Step 2 and Activity 3), this tool helps the EQRO review
the:
•

Accuracy of data transfers to the assigned performance measure repository

•

Accuracy of file consolidations, extracts, and derivations

•

Adequacy of the performance measure data repository to calculate and report performance measures, and

•

Management of report production and reporting software

For each data integration and control element, please indicate whether it was met, not met, or not applicable (N/A),
and any relevant comments.
1. Accuracy of data transfers to assigned
performance measure repository

Met

Not
met

N/A

Comments

Met

Not
met

N/A

Comments

MCP processes accurately and completely
transfer data from the transaction files (e.g.,
membership, provider, encounter/claims) into the
repository used to keep the data until the
calculations of the performance measures have
been completed and validated
Samples of data from repository are complete and
accurate

2. Accuracy of file consolidations, extracts, and
derivations
MCP’s processes to consolidate diversified files
and to extract required information from the
performance measure repository are appropriate
Actual results of file consolidations or extracts
were consistent with those which should have
resulted according to documented algorithms or
specifications
Procedures for coordinating the activities of
vendors ensure the accurate, timely, and
complete integration of data into the performance
measure database
Computer program reports or documentation
reflect vendor coordination activities, and no data
elements needed for performance measure
reporting are lost or inappropriately modified
during transfer

PROTOCOL TWO | 95

3. If the MCP uses one, the structure and format of
the performance measure data repository
facilitates any required programming necessary to
calculate and report required performance
measures

Met

Not
met

N/A

Comments

Met

Not
met

N/A

Comments

The repository’s design, program flow charts, and
source codes enable analyses and reports
Proper linkage mechanisms have been employed
to join data from all necessary sources (e.g.,
identifying a member with a given
disease/condition)

4. Assurance of effective management of report
production and of the reporting software
Documentation governing the production process,
including MCP production activity logs, and MCP
staff review of report runs was adequate
Prescribed data cutoff dates were followed
The MCP has retained copies of files or
databases used for performance measure
reporting, in the event that results need to be
reproduced
Reporting software program is properly
documented with respect to every aspect of the
performance measurement reporting repository,
including building, maintaining, managing, testing,
and report production
MCP’s processes and documentation comply with
the MCP standards associated with reporting
program specifications, code review, and testing

96 | PROTOCOL TWO

Worksheet 2.7. Data and Processes Used to Produce Performance Measures:
Documentation Review Checklist
Instructions. During the onsite visit, this tool helps the EQRO check the documentation of steps taken in the
production of the performance measures. It is intended to guide document review (as described in Activity 2, Step 3).

Documentation

Reviewed

Not
Reviewed

Not
applicable

Comments

Documentation of overall policies and
procedures
Policies which stipulate and enforce
documentation of data requirements, issues,
validation efforts and results
Procedures for displaying denominator
counts, numerator counts, precision levels,
and totals
Procedures for reporting small sample sizes
(consistent with state’s required methodology)
All reported measures to assess consistency
of common elements (e.g., membership
counts, number of pregnancies and births,
etc.)
Documentation for each measure:
Programming logic and/or source code for
arithmetic calculation
A project or measurement plan, including
work flow
Documentation of programming specifications
and data sources
Documentation of the original universe of data
including record-level patient identifiers that
can be used to validate entire programming
logic for creating denominators, numerators,
and samples
Documentation of computer queries,
programming logic, or source code used to
create denominators, numerators, and interim
data files
Documentation of medical record review for
each measure, as appropriate, including:
qualifications of medical record review
supervisor and staff; reviewer training
materials, audit tools used (including
completed copies of each record-level
reviewer determination), all case-level critical
performance measure data elements used to
determine a positive or negative event or
exclude a case from same, and inter-rater
reliability testing procedures and results
Documentation of results of statistical tests
and any corrections or adjustments to data
along with justification for such changes for
each measure, as appropriate
Documentation showing calculation of levels
of significance of changes for each measure

PROTOCOL TWO | 97

Documentation
Documentation showing confidence intervals
of calculations when sampling methodology
used
Documentation of sources of any supporting
external data or prior years’ data used in
reporting
Overall assessment: In the comments
section, note:
• How are policies governing documentation
of data requirements for performance
measurement (e.g., data file and field
definitions, mapping between standard and
non-standard codes) updated and
enforced? Who is responsible for this?
• How are programming specifications for
MCP performance measures documented?
Who is responsible for this?
• Are documentation processes up to date?

98 | PROTOCOL TWO

Reviewed

Not
Reviewed

Not
applicable

Comments

Worksheet 2.8. Data and Processes Used to Produce Performance Measures: Findings
Instructions. Record findings based on measurement plans, policies, and programming specifications, as described
in Activity 2, Step 3.
1. Measurement plans and policies that stipulate and
enforce documentation of data requirements, issues,
validation efforts, and results. These include the
following audit elements

Met

Not
Met

N/A

Comments

Data file and field definitions used for each measure
Maps to standard coding if not used in original data
collection
Statistical testing of results and any corrections or
adjustments made after processing
Overall assessment: In the comments section, note
any recommendations for improving measurement plans
and policies

PROTOCOL TWO | 99

2. Documentation of programming specifications (which
may be either a schematic diagram or in narrative form)
for each measure includes at least the following audit
elements:
All data sources, including external, supplemental data
(whether from a vendor, public registry, or other outside
source), and any prior year data, if applicable
A project or measurement plan, including workflow
Detailed medical record review methods and practices,
including:
• The qualifications of medical record review supervisor
and staff
• Reviewer training materials
• Audit tools used (including completed copies of each
record-level reviewer determination)
• All case-level critical performance measure data
elements used to determine a positive or negative
event or exclude a case from same, and
• Inter-rater reliability testing procedures and results
Detailed computer queries, programming logic, or source
code used to identify the population or sample for the
denominator and/or numerator
Documentation of the original universe of data including
record-level patient identifiers that can be used to
validate entire programming logic for creating
denominators, numerators, and samples
If sampling is used, a description of sampling techniques
and documentation assuring the reviewer that samples
used for baseline and repeat measurements of the
performance measures were chosen using the same
sampling frame and methodology
Documentation of calculation for changes in
performance from previous periods, if applicable,
including statistical tests of significance
Data that are related from measure to measure are
consistent (e.g., membership counts, provider totals,
number pregnancies and births)
Appropriate statistical functions are used to determine
confidence intervals when sampling is used in the
measure
When determining improvement in performance between
measurement periods, appropriate statistical
methodology is applied to determine levels of
significance of changes
Overall assessment: In the comments section, note
any recommendations for improving programming
specifications for each performance measure

100 | PROTOCOL TWO

Met

Not
Met

N/A

Comments

Worksheet 2.9. Policies, Procedures, and Data Used to Produce Performance Measures:
Review Checklist
Instructions. Use this checklist to track documents and data used to assess the accuracy of the MCP’s performance
measure calculations (as described in Activity 2, Step 4).

Policies, Procedures, and Data to be Reviewed

Reviewed

Not
Reviewed

Comments

Policies to assign unique membership ID that allows
all services to be properly related to the specific
appropriate enrollee, despite changes in status,
periods of enrollment or disenrollment, or changes
across product lines (e.g., Medicare and Medicaid)
Procedures to identify, track, and link member
enrollment by product line, product, geographic area,
age, gender, member months, member years
Procedures to track individual members through
enrollment, disenrollment, and possible re-enrollment
Procedures to track members through changes in
family status, changes in employment or benefits or
managed care type (if they switch between Medicaid
coverage and another product within the same MCP)
Methods to define start and cessation of coverage
Procedures to link member months to member age
Description of software or programming languages
used to query each database
Programming logic and/or source code for arithmetic
calculation of each measure
Programming logic and/or source code for measures
with complex algorithms, to ensure adequate matching
and linkage among different types of data
Member database
Provider data (including facilities, labs, pharmacies,
physicians, etc.)
Database record layout and data dictionary
Survey data
For performance measures which are easily underreported, procedures to capture data that may reside
outside the MCP’s data sets
Procedures for mapping non-standard codes to
standard coding to ensure consistency, completeness,
and reproducibility
Policies, procedures, and materials that evidence
proper training, supervision, and adequate tools for
medical record abstraction tasks (may include medical
record abstraction tools, training material, checks of
inter-rater reliability, etc.)
Procedures for assuring that combinations of recordreview data with administratively determined data are
consistent and verifiable

PROTOCOL TWO | 101

Policies, Procedures, and Data to be Reviewed
Evidence that MCP’s use of codes to identify medical
events were correctly evaluated when classifying
members for inclusion or exclusion in the numerator
Evidence that MCP has counted each member and/or
event only once
Programming logic or demonstration that confirms that
any non-standard codes used in determining the
numerator have been mapped to a standard coding
scheme in a manner that is consistent, complete, and
reproducible
Programming logic or source code that identifies
process for integrating administrative and medical
record data for numerator
Programming logic and/or source code for arithmetic
calculation of each measure
Programming logic and/or source code for measures
with complex algorithms, to ensure adequate matching
and linkage among different types of data
Overall assessment: In the comments section, note
any recommendations to improve documentation or
demonstrations provided by the MCP

102 | PROTOCOL TWO

Reviewed

Not
Reviewed

Comments

Worksheet 2.10. Measure Validation Findings
Instructions. For each performance measure, the EQRO can use this worksheet to document adherence to
guidance for (1) the denominator; (2) programming logic, source code, and calculations; (3) identifying medical
events; (4) time parameters; (5) exclusion criteria; (6) population estimates; (7) identifying the at-risk population; (8),
inclusion of qualifying medical events in the numerator; and (9) medical record data in the numerator. This worksheet
is relevant to Activity 2, Step 4, and Activity 3.

Audit Element

Met

Not
Met

N/A

Comments

Denominator. For each performance measure, all
members of the relevant populations identified in the
performance measure specifications (who were
eligible to receive the specified services) were
included in the population from which the denominator
was produced. The eligible population included
members who received the services as well as those
who did not. The same standard applies to provider
groups or other relevant populations identified in the
specifications of each performance measure.
Programming logic, source code, and
calculations. For each measure, adequate
programming logic or source code identifies, tracks,
and links member enrollment within and across
product lines (e.g., Medicaid and CHIP), by age and
sex, as well as through possible periods of enrollment
and disenrollment) and appropriately identifies all
relevant members of the specified denominator
population for each of the performance measures.
This is determined by evaluating that:
1. Calculations of continuous enrollment criteria were
correctly carried out and applied to each measure
(if applicable)
2

Proper mathematical operations were used to
determine patient age or age range

3. The MCP can identify the variable(s) that define
the member’s sex in every file or algorithm needed
to calculate the performance measure
denominator, and the MCP can explain what
classification is carried out if neither of the
required codes is present
4. The MCP has correctly calculated member
months and member years, if applicable to the
performance measure
Identifying medical events. The MCP has properly
evaluated the completeness and accuracy of any
codes used to identify medical events, such as
diagnoses, procedures, or prescriptions, and these
codes have been appropriately identified and applied
as specified in each performance measure.
Time parameters. Any time parameters required by
the performance measure specification were followed
by the MCP (e.g., cut off dates for data collection,
counting 30 calendar days after discharge from a
hospital).

PROTOCOL TWO | 103

Audit Element
Exclusion criteria. Performance measure
specifications or definitions that exclude members
from a denominator were followed. (For example, if a
measure relates to receipt of a specific service, the
denominator may need to be adjusted to reflect
instances in which the patient refuses the service or
the service is contraindicated.)
Population estimates. Systems or methods used by
the MCP to estimate populations when they cannot be
accurately or completely counted (e.g., newborns) are
valid.
Identifying the at-risk population. The MCP has
used the appropriate data, including linked data from
separate data sets, to identify the entire at-risk
population.
Services provided outside the MCP. The MCP has
adopted and followed procedures to capture data for
those performance measures that could be easily
under-reported due to the availability of services
outside the MCP. (For some measures, particularly
those focused on women and children, the member
may have received the specified service outside of
the MCP provider base, such as children receiving
immunizations through public health services or
schools, access to family planning services. An extra
effort must be made to include these events in the
numerator.)
Inclusion of qualifying medical events. The MCP’s
use of codes to identify medical events (e.g.,
diagnoses, procedures, prescriptions) are complete,
accurate, and specific in correctly describing what
transpired and when. This included:
1, The MCP correctly evaluated medical event codes
when classifying members for inclusion or
exclusion in the numerator
2

The MCP avoided or eliminated all doublecounted members or numerator events

3. The MCP mapped any non-standard codes used
in determining the numerator in a manner that is
consistent, complete, and reproducible. The
EQRO assesses this through a review of the
programming logic or a demonstration of the
program
4. Any time parameters required by the
specifications of the performance measure were
adhered to (i.e., that the measured event occurred
during the time period specified or defined in the
performance measure)
Medical record data. Medical record reviews and
abstractions were carried out in a manner that
facilitated the collection of complete, accurate, and
valid data by ensuring that:
1. Record review staff have been properly trained
and supervised for the task
2. Record abstraction tools required the appropriate
notation that the measured event occurred

104 | PROTOCOL TWO

Met

Not
Met

N/A

Comments

Audit Element

Met

Not
Met

N/A

Comments

3. Record abstraction tools required notation of the
results or findings of the measured event, if
applicable
4. Medical record data from electronic sources was
accurately extracted according to measure
specifications
5. Data included in the record extract files are
consistent with data found in the medical records
based on a review of a sample of medical record
for applicable performance measures
6. The process of integrating administrative data and
medical record data for the purpose of determining
the numerator is consistent and valid
Overall assessment: In the comments section, note
any recommendations (if applicable) to:
• Improve the denominator
• Improve programming logic, source code, or
calculations
• Improve the completeness or accuracy of the
codes used to identify medical events
• Improve the specified time parameters
• Improve adherence to the exclusion criteria
• Improve systems/methods to estimate populations
when they cannot be accurately counted
• Ensure all appropriate data are used to identify the
entire at-risk population
• Appropriately identify and include qualifying
medical events for the numerator
• Improve the proper collection of medical record
data extracted for inclusion in the numerator

PROTOCOL TWO | 105

Worksheet 2.11. Interview Guide for Assessing Processes and Procedures Used to
Produce Numerators and Denominators
Instructions. The following interview guide may be used to supplement findings reported in Worksheet 2.10 (as
described in Activity 2, Step 4). Please tailor the questions as appropriate.
MCP Contact and Background Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, email address, and date of interview, if applicable.
MCP name:
MCP contact name(s):
Title(s):
Mailing address:
Phone number(s):
Email address:
Interview Date:
EQRO reviewers:
Year of first Medicaid enrollment:
Year of first CHIP enrollment:
Year of first MCP performance report
(any product line)

1. If any part of your network/data/membership was excluded from a performance measure, how and why did you
decide to exclude it?
2. Why did you select the reporting methodology (e.g., administrative, or hybrid) used to create each of the
measures (where there was an option)?
3. Did you use the state technical specifications as the specifications for the programmers, or did your MCP write its
own instructions/translations for the programmers?
4. Are there any manual processes used for calculating denominators and/or numerators? Are manual processes
used for sampling?
5. Are any measures calculated by vendors? If yes, are they checked for accuracy? Please describe.
6. Do you have any concerns about the integrity of the information used to create any of the measures? Please
describe.
7. Do you know of any deviations from performance measure specifications that were necessary because of data
available or because of your MCP’s information system capabilities?
8. Other issues.

106 | PROTOCOL TWO

Worksheet 2.12. Policies, Procedures, and Data Used to Implement Sampling: Review
Checklist
Instructions. This checklist provides a list of documents, procedures, and data to assess the sampling process, if
applicable (as described in Activity 2, Step 5).

Documents

Reviewed

Not
Reviewed

Comments

Description of software used to execute sampling sort
of population files when sampling (e.g., systematic) is
used
Policies to maintain files from which the samples are
drawn in order to keep population intact in the event
that a sample must be re-drawn or replacements
made
Computer source code or logic identifying specified
sampling techniques, and documentation that the
logic matches the specifications set forth for each
performance measure, including sample size and
exclusion methodology
Methods used for sampling for measures calling for
hybrid data or medical record review
Documentation assuring that sampling methodology
treats all measures independently, and that there is
no correlation between drawn samples
Observation of or documentation of procedures in
which a biased sample was identified and corrected
Documentation of “frozen” or archived files from
which the samples were drawn, and if applicable,
documentation of the MCO’s process to re-draw a
sample or obtain necessary replacements
Overall assessment: In the comments section, note
any recommendations to improve documentation or
demonstrations to assess the sampling process

PROTOCOL TWO | 107

Worksheet 2.13. Sampling Validation Findings
Instructions. This checklist provides a list of documents, procedures, and data to assess the sampling process
across the following elements: (1) the MCP followed the specified sampling method to produce an unbiased sample
that is representative of the entire included population, (2) the MCP maintains its performance measurement
population files/data sets in a manner that allows a sample to be re-drawn or used as a source for replacement, (3)
sample sizes collected conform to the methodology set forth in the performance measure specifications and the
sample is representative of the entire population, and (4) for performance measures that include medical record
review (e.g., hybrid data collection methodology), proper substitution methodology was followed. This worksheet is
applicable to Activity 2, Step 4 and Activity 3.

1. Audit Element: Sampling method

Met

Not
Met

N/A

Comments

Met

Not
Met

N/A

Comments

Each relevant member or provider had an equal
chance of being selected; no one was systematically
excluded from the sampling.
The MCP followed the specifications set forth in the
performance measure regarding the treatment of
sample exclusions and replacements, and if any
activity took place involving replacements of or
exclusions from the sample, the MCP kept adequate
documentation of that activity
Each provider serving a given number of enrollees
had the same probability of being selected as any
other provider serving the same number of enrollees
The MCP examined its sampled files for bias, and if
any bias was detected, the MCP is able to provide
documentation that describes any efforts taken to
correct it
The sampling methodology employed treated all
measures independently, and there is no correlation
between drawn samples
Relevant members or providers who were not
included in the sample for the baseline measurement
had the same chance of being selected for the
follow-up measurement as providers who were
included in the baseline
Overall assessment: In the comments section, note
any recommendations to produce an unbiased,
representative sample

2. Audit Element: Performance measurement
files/data
The MCP has policies and procedures to maintain
files from which the samples are drawn in order to
keep the population intact in the event that a sample
must be re-drawn, or replacements made, and
documentation that the original population is intact
Overall assessment: In the comments section, note
any recommendations to improve file or data
maintenance

108 | PROTOCOL TWO

3. Audit Element: Performance measure
specifications

Met

Not
Met

N/A

Comments

Met

Not
Met

N/A

Comments

Sample sizes meet the requirements of the
performance measure specifications
The MCP has appropriately handled the
documentation and reporting of the measure if the
requested sample size exceeds the population size
The MCP properly oversampled in order to
accommodate potential exclusions
Overall assessment: In the comments section, note
any recommendations to improve adherence to
performance measure specifications

4. Audit Element: Medical record reviews
Substitution applied only to those members who met
the exclusion criteria specified in the performance
measure definitions or requirements
Substitutions were made for properly excluded
records and the percentage of substituted records
was documented
Overall assessment: In the comments section, note
any recommendations to improve use of proper
substitution methodology

PROTOCOL TWO | 109

Worksheet 2.14. Framework for Summarizing Information About Performance Measures
Instructions. Use this worksheet or a similar tool to summarize the results for each performance measure validated
for each managed care plan. This worksheet can be used as a framework for summarizing validation at the plan
level. In addition, the information in this worksheet can be aggregated across plans and measures to generate
information on state-level performance and areas for improvement.

1. Overview of Performance Measure
Managed Care Plan (MCP) name:
Performance measure name:
Measure steward:
Agency for Healthcare Research and Quality (AHRQ)
Centers for Disease Control and Prevention (CDC)
Centers for Medicare & Medicaid Services (CMS)
National Committee for Quality Assurance (NCQA)
The Joint Commission (TJC)
No measure steward, developed by state/EQRO
Other measure steward (specify) _____________________________________________
Is the performance measure part of an existing measure set? (check all that apply)
HEDIS®
CMS Child or Adult Core Set
Other (specify) ____________________________________________
What data source(s) was used to calculate the measure? (check all that apply)
Administrative data (describe) ________________________________
Medical records (describe) __________________________________
Other (specify) ____________________________________________
If the hybrid method was used, describe the sampling approach used to select the medical records:
Not applicable (hybrid method not used)
Definition of denominator (describe):
Definition of numerator (describe):
Program(s) included in the measure:
CHIP
Measurement period (start/end date)

110 | PROTOCOL TWO

Medicaid (Title XIX) only

CHIP (Title XXI) only

Medicaid and

2. Performance Measure Results (If measure contains more than one rate, add columns to the
table)

Performance
measure

Rate 1

Rate 2

Rate 3

Rate 4

Numerator
Denominator
Rate

Describe any findings from the ISCA or other information systems audit that affected the reliability or validity of the
performance measure results.
Not applicable (ISCA not reviewed)
Describe any findings from medical record review that affected the reliability or validity of the performance
measure results.
Not applicable (medical record review not conducted)
Describe any other validation findings that affected the accuracy of the performance measure calculation.

PROTOCOL TWO | 111

Example of Worksheet 2.14. Framework for Summarizing Information about Performance
Measures
Instructions. Use this worksheet or a similar tool to summarize the results for each performance measure validated
for each managed care plan. This worksheet can be used as a framework for summarizing validation at the plan
level. In addition, the information in this worksheet can be aggregated across plans and measures to generate
information on state-level performance and areas for improvement.

1. Overview of Performance Measure
Managed Care Plan (MCP) name: Plan A
Performance measure name: Follow-Up After Hospitalization for Mental Illness: 6-20 (FUH-CH)
Measure steward:
Agency for Healthcare Research and Quality (AHRQ)
Centers for Disease Control and Prevention (CDC)
Centers for Medicare & Medicaid Services (CMS)
National Committee for Quality Assurance (NCQA)
The Joint Commission (TJC)
No measure steward, developed by state/EQRO
Other measure steward (specify) _____________________________________________
Is the performance measure part of an existing measure set? (check all that apply)
HEDIS®
CMS Child or Adult Core Set
Other (specify) ____________________________________________
What data source(s) was used to calculate the measure? (check all that apply)
Administrative data (describe): The administrative data source is the state’s MMIS and data submitted by the
managed care plans.
Medical records (describe) __________________________________
Other (specify) ____________________________________________
If the hybrid method was used, describe the sampling approach used to select the medical records:
Not applicable (hybrid method not used)
Definition of denominator (describe): Medicaid rates include managed care population (4 MCOs) age 6 and older.
Definition of numerator (describe):
•

7-day follow-up: A follow-up visit with a mental health practitioner within 7 days after discharge. This
includes visits that occur on the date of discharge.

•

30-day follow-up: A follow-up visit with a mental health practitioner within 30 days after discharge. This
includes visits that occur on the date of discharge.

Program(s) included in the measure:
CHIP

Medicaid (Title XIX) only

CHIP (Title XXI) only

Measurement period (start/end date): January 1, 2015 to December 31, 2015.

112 | PROTOCOL TWO

Medicaid and

2. Performance Measure Results (If measure contains more than one rate, add columns to the
table)

Performance
measure

Rate 1

Rate 2

Numerator
Denominator
Rate

Rate 3

6,723

8,476

12,007

56.0
(7-day follow-up)

70.6
(30-day follow-up)

Rate 4

3. Performance Measure Validation Status
Describe any deviations from the technical specifications and explain reasons for deviations (such as deviations in
denominator, numerator, data source, measurement period, or other aspect of the measure calculation).
Amerigroup was compliant with the HEDIS® Information System Standards and HEDIS® Determination
Standards, and continues to use NCQA-certified software vendors for HEDIS® measure production.
Describe any findings from the ISCA or other information systems audit that affected the reliability or validity of the
performance measure results.
Not applicable (ISCA not reviewed)
Describe any findings from medical record review that affected the reliability or validity of the performance
measure results.
Not applicable (medical record review not conducted)
Describe any other validation findings that affected the accuracy of the performance measure calculation.
No findings to report.

Validation rating:
High confidence
Moderate confidence
Low confidence
No confidence
“Validation rating” refers to the EQRO’s overall confidence that the calculation of the performance measure
adhered to acceptable methodology.
EQRO recommendations for improvement of performance measure calculation:
The FUH-CH measure represents one of the objectives in the state’s Quality Strategy (e.g., child health,
prevention, and screening services), which seeks to assure timely, high-quality health care for all [State Medicaid
Program Name] members. The EQRO has no recommendations to improve the performance measure calculation.

END OF WORKSHEETS FOR PROTOCOL 2

PROTOCOL TWO | 113

This page left blank for double-sided copying.

114 | PROTOCOL TWO

PROTOCOL 3. REVIEW OF COMPLIANCE WITH
MEDICAID AND CHIP MANAGED CARE
REGULATIONS
A MANDATORY EQR-RELATED ACTIVITY

ACTIVITY 2: PERFORM THE PRELIMINARY REVIEW (PRE-ONSITE
VISIT)
ACTIVITY 3: CONDUCT MCP ONSITE VISIT
ACTIVITY 4: COMPILE AND ANALYZE FINDINGS (POST-ONSITE VISIT)
ACTIVITY 5: REPORT RESULTS TO THE STATE

BACKGROUND
This protocol is used to determine the extent to which Medicaid and CHIP
managed care plans (MCPs) are in compliance with federal standards.
The Department of Health & Human Services (HHS) developed standards
for managed care plans (MCPs), which are codified at 42 C.F.R. § 438
and 42 C.F.R. § 457, as revised by the Medicaid and CHIP managed care
final rule issued in 2016. As noted in the Introduction, states have the
option to use information from a Medicare or private accreditation review
of an MCP to provide information for the annual EQR instead of
conducting this mandatory EQR-related activity. 54, 55

If the state elects to use nonduplication for this mandatory EQR-related activity (42 C.F.R. §
438.360, Nonduplication of mandatory activities with Medicare or accreditation review), then the
state must ensure that all information from the Medicare or private accreditation review is
provided to the EQRO for analysis and inclusion in the annual EQR technical report. (See 42
C.F.R. § 438.360(a)(1)–(3) for additional details regarding the circumstance under which
nonduplication is an option). Use of nonduplication must be identified in the state’s quality
strategy (see 42 C.F.R. § 438.360(c) and 438.340(b)(10)). Any requirements in this protocol
which are not addressed via the review used for nonduplication must still be addressed through
this protocol. CHIP cross-references to this requirement at §457.1250, but does not allow for
the use of Medicare review activities for the purposes of nonduplication.

PROTOCOL THREE

ACTIVITY 1: ESTABLISH COMPLIANCE THRESHOLDS

A state may not utilize nonduplication if Medicare has accepted an only attestation of a plan’s
QIP. In the context of this EQR-related activity, the QIP would have to undergo validation as
part of a Medicare review in order for nonduplication to be an option. See 42 C.F.R. §
438.360(a)(2).
55

115

Regulations Subject to Compliance Review
The standards that are the subject of this protocol are contained in 42 C.F.R. 438, Subparts D
and E. 56 The scope of those sections includes: 57
•

Availability of services § 438.206

•

Assurances of adequate capacity and services § 438.207

•

Coordination and continuity of care § 438.208

•

Coverage and authorization of services § 438.210

•

Provider selection § 438.214

•

Confidentiality § 438.224

•

Grievance and appeal systems § 438.228

•

Subcontractual relationships and delegation § 438.230

•

Practice guidelines § 438.236

•

Health information systems § 438.242

•

Quality assessment and performance improvement program § 438.330

Additional Areas for Potential Compliance Review 58
CMS encourages states to consider expanding the scope of the review to cover compliance with
federal and state requirements beyond those specified in 42 C.F.R. § 438, including other state
statutory, regulatory, or contractual requirements related to the following areas, if applicable:
•

Accessibility, including physical accessibility of service sites and medical and diagnostic
equipment; accessibility of information (compliance with web-based information, literacy
levels of written materials, and alternate formats); and other accommodations. See Section
508 of the Rehabilitation Act [29 U.S.C. § 794d])

•

Availability and use of home- and community-based services (HCBS) as alternatives to
institutional care, so individuals can receive the services they need in the most integrated
setting appropriate

•

Credentialing or other selection processes for long-term services and supports (LTSS)
providers, including those required where the enrollee can choose their caregiver (such as
verification of completion of criminal background checks)

•

Person-centered assessment, person-centered care planning, service planning and
authorization, service coordination and care management for LTSS, including
authorization/utilization management for LTSS and any beneficiary rights or protections

CHIP cross-references to these requirements at 42 C.F.R. § 457.1230, 457.1233, and 457.1240, except as noted. For
more information, see https://www.federalregister.gov/documents/2016/05/06/2016-09581/medicaid-and-childrens-healthinsurance-program-chip-programs-medicaid-managed-care-chip-delivered#sectno-citation-%E2%80%89438.206.
57
Certain requirements in Subparts A, B, C, and F are incorporated into the compliance review through interaction with
Subparts D and E.
58
For more information, see the CMS MLTSS EQR guidance document at https://www.medicaid.gov/medicaid-chipprogram-information/by-topics/delivery-systems/downloads/cmcs-eqr-protocols.pdf.
56

116 | PROTOCOL THREE

related to care planning and service planning such as conflict-free case management, selfdirection of services, and appeal rights related to person-centered planning
•

Integration of managed medical, behavioral, and LTSS

Frequency of Compliance Review and Manner of Reporting
Federal regulations require MCPs to undergo a review at least once every three years to
determine MCP compliance with federal standards as implemented by the state. States may
choose to direct their EQROs to review all applicable standards at once or may spread the
review over a three-year cycle in any manner they choose (for example, fully reviewing a third of
plans each year or conducting a third of the review on all plans each year). However, if an EQR
technical report summarizes a compliance review that does not include all required
components, the report should clearly describe:
1

The three-year period covered by the current compliance review cycle

The quality standards not included in the current report

A summary of findings from all previous reviews within the current review cycle

The state’s schedule for review of the remaining standards

GETTING STARTED ON PROTOCOL 3
This protocol describes the process EQROs may use to determine MCP compliance with
federal Medicaid and CHIP managed care regulations. In general, the EQRO must:
1

Review program documents and conduct interviews with MCP personnel to collect
information, and

Analyze information collected and make compliance determinations

To complete this protocol, the EQRO must undertake five activities for each MCP (Figure 3.1).

PROTOCOL THREE | 117

Figure 3.1. Protocol 3 Activities

One supplement resource is available to help EQROs conduct the compliance review:
•

Worksheets for Protocol 3. Compliance Review Tools, which can be used to 1) structure
and conduct the review of MCP documentation to determine compliance with the applicable
federal regulatory and/or state provisions; 2) score MCPs’ compliance with federal and
state regulations; 3) develop onsite visit agendas, and 4) guide onsite compliance
interviews of MCP staff

The remainder of this protocol outlines the steps associated with Activities 1 through 5.

118 | PROTOCOL THREE

ACTIVITY 1: ESTABLISH COMPLIANCE THRESHOLDS
In this activity, the EQRO works with the state to define levels of compliance for use throughout
the compliance review. Activity 1 includes two steps:
1

Determine how the state has implemented federal quality standards

Define the level of compliance against which the MCP will be measured or scored

Step 1: Collect Information from the State

WORKSHEET 3.1

Some regulatory provisions allow the state to
establish more stringent standards for their MCPs
than are mandated by federal regulation. Additional
state requirements may be found in MCP contracts,
state managed care quality strategies, state statutes
and regulations, or other resources. Therefore, the
EQRO will need to know the state’s requirements for
its MCPs in order to complete the compliance
assessment. This documentation may be provided
to the EQRO in hard copy, digital copy, or both
formats. Worksheet 3.1 contains the types of
documents that the state may provide to the EQRO
about state standards and is organized according to
federal regulatory provision.

Resources for Activity 1, Step 1
Worksheet 3.1. Compliance Review
• Includes a list of the types of state
policy/regulation information needed to
determine MCP compliance, as well as
documents the MCP may provide the
EQRO to demonstrate the MCP's
compliance with federal regulations and
state standards (see Activity 2, Step 2)
• The completed Compliance Review
Worksheet is a primary data source for
analyses and a comprehensive record of
compliance protocol EQR-related activities

Step 2: Define Levels of Compliance

WORKSHEET 3.2

EQRO determinations will be based on compliance
definitions set in advance by the state for each
federal and/or state regulatory provision, component
of a provision, and/or requirement or standard
based on its expectations of MCP performance.
While states may define multiple levels of
compliance, a definition for full compliance must be
clearly understood by the EQRO and MCP before
the review.

Resources for Activity 1, Step 2
Worksheet 3.2. Compliance Definitions
• Provides compliance definitions and
examples of compliance rating scales
• Compliance rating scales may be adjusted
to best suit a state’s needs

ACTIVITY 2: PERFORM THE PRELIMINARY REVIEW (PRE-ONSITE
VISIT)
Site visits are an effective way to collect the information needed for quality oversight and
compliance determination. However, they require careful planning to maximize the information
obtained and to minimize the time required for collecting that information. This activity should
begin from 2 to 6 months in advance of the planned visit.

PROTOCOL THREE | 119

Step 1: Establish Early Contact with the MCP

It is important for the EQRO to establish and maintain consistent communication with the MCP
throughout the compliance review. The EQRO is responsible for developing a communication
plan specifying expectations for all parties involved. The EQRO should establish a single point
of contact with the MCP, and in turn, the MCP representative can organize the response from
the MCP and determine which additional staff members should be involved during the review.
Step 2: Perform a Document Review

WORKSHEET 3.1

The purpose of the document review is to identify gaps
in information to ensure a comprehensive EQR process
and productive interactions with the MCP during the
onsite visit. Before planning an onsite visit, the EQRO
should gather and assess as much information about the
MCP and its practices as possible. Document review
includes gathering information about the MCP’s
background, including its structure, enrolled population,
providers, services, resources, locations, delegated
functions and services, and contractors. Some
information may be available from the state, while some
may be obtained from the MCP. The following list
suggests the type of information that would be useful
during a preliminary document review:

Resources for Activity 2, Step 2
Worksheet 3.1. Compliance Review
• Includes a list of the types of state
policy or regulatory information
needed to determine MCP
compliance (see Activity 1, Step 1),
as well as documents the MCP may
provide the EQRO to demonstrate
the MCP's compliance with federal
regulations and state standards
• The completed Compliance Review
Worksheet is a primary data source
for analyses and a comprehensive
record of compliance protocol EQRrelated activities

•

Organization name and mailing address

•

Contact person’s name, title, phone number, and email address

•

Site visit location

•

Organizational charts or other descriptions of the MCP

•

Product lines offered

•

Total individuals enrolled in the current and previous year

•

Total number of network practitioners in the current and previous year, with a breakdown by
type (such as primary care, OB/GYN, and other specialties)

•

Total number of network organizational providers (hospitals, ambulatory care, home care,
laboratories, etc.)

•

Service descriptions and benefit designs available to enrollees

•

Delegated activities, including MCP subcontractors

•

Data on the MCP Quality of Care review

The EQRO should inform the MCP of any missing information prior to the onsite visit, to allow
the MCP to respond in a timely manner (either by providing the documentation to the EQRO
prior to the onsite visit for review or ensuring its availability during the onsite visit document
review, see Activity 3 Step 5). The EQRO should maintain consistent documentation by adding
120 | PROTOCOL THREE

preliminary document review findings or questions for follow-up during the onsite visit to
Worksheet 3.1. 59

ACTIVITY 3: CONDUCT MCP ONSITE VISIT
The purpose of the MCP onsite visit is to collect the information necessary to assess the MCP’s
compliance with federal and state regulations through additional document review and onsite
interviews. The EQRO should plan the onsite visit in accordance with the compliance review
plan established in Activity 1 of this protocol. As noted in Activity 2, the EQRO should review
MCP policies and procedures before the onsite visit to expedite the process.
Pre-Onsite Visit Preparation

Steps 1 through 5 provide guidance in preparing for the onsite visit.
Step 1: Determine Onsite Visit Length and Dates

The length of a comprehensive onsite visit will vary according to the scope of the review, the
complexity of the organization being reviewed, the number of reviewers available to conduct the
review, and the amount of information collected before the onsite visit. A typical onsite visit
requires 3 to 5 days. To schedule the onsite visit, the EQRO should offer the MCP contact a
range of dates to determine when essential staff are available.
Step 2: Identify the Number and Types of Reviewers Needed

Reviewers should be skilled interviewers with the ability to read and process a variety of data in
order to determine whether an MCP is in compliance with the regulations. Knowledge or
experience in state Medicaid/CHIP programs and managed care is highly desirable. Reviewer
orientation and training should be held to ensure familiarity with the regulatory provisions, the
evaluation process, and performance expectations.
The number of reviewers needed to conduct the onsite assessment should be based on the
characteristics of the MCP being evaluated. Consideration should be given to the size and
complexity of the MCP, including the size of the provider network, number of enrollees, and the
scope of programs in the state contract. If multiple reviewers are participating in the onsite
visit(s), the EQRO should identify in advance each reviewer’s responsibility for assessing
specific standards, reviewing specific documents, and conducting interviews.
Step 3: Develop an Onsite Visit Agenda

WORKSHEET 3.3

In addition to the document review described here as part of the preliminary document review, the previous version of this
protocol included a step in Activity 3 (Conduct Onsite MCP Visit) for document review while onsite. However, due to the
widespread use of digital documentation, an additional document review conducted during the onsite review is not expected
to generally be needed. If an onsite document review is determined necessary by the EQRO (and negotiated with the MCP),
the EQRO may conduct it. That step would occur prior to Activity 3, Step 5.

PROTOCOL THREE | 121

Clear expectations are essential for an efficient and
effective onsite visit. An agenda sets the expectations
and schedule for the review. It also assists both the
MCP and EQRO in planning staff participation,
gathering documentation, and finalizing logistics, such
as arranging locations for document review and
interviews. The EQRO should consult with the MCP
throughout the agenda setting process to ensure the
inclusion of appropriate staff.

Resources for Activity 3, Step 3
Worksheet 3.3. Sample Onsite Visit Agenda
• This template can be used to develop the
onsite visit agenda. It is intended to help
the MCP and EQRO in planning staff
interviews, gathering documents, and
finalizing logistics

Step 4: Provide Preparation Instructions and Guidance to the MCP

The EQRO should send clear instructions and guidance to the MCP before the onsite visit. In
preparation for the onsite visit, the EQRO should provide MCPs with the following information:
1

The scope of the assessment

How the review will be conducted

List of required documents

Instructions for how documents for review should be organized

Forms or other data gathering instruments that should be completed before arrival (such as
the Information Systems Capability Assessment (ISCA); see Appendix A)

Reports from prior reviews and subsequent MCP corrective actions

Names and contact information for expected interview participants, and

Administrative needs of the reviewers

Onsite Visit Activities
Step 5: MCP Interviews

WORKSHEET 3.1
WORKSHEET 3.4

The purpose of MCP interviews is to collect data to supplement and verify what is learned
through document review. In preparation for the onsite visit, the EQRO should review the
standards identified in the state documents obtained in Activity 1 and the findings from Activity
2, Step 2. During the onsite visit, MCP staff should be available if the EQRO has questions or
difficulty locating any needed additional documents or other information. The EQRO should
notify the MCP during the onsite visit of any missing information to allow the MCP to respond in
a timely manner. The EQRO should maintain consistent documentation by adding to Worksheet
3.1 any findings based on any additional information or documents provided by the MCP.

122 | PROTOCOL THREE

Prepare for the Interviews
Interviews should be tailored to the MCP being
evaluated and the role of the interviewee. When
planning for the interview, the EQRO should:
•

•

Prepare a list of issues to be addressed in
each interview, based on federal regulatory
provisions, state standards, MCP
organization characteristics, and other
information gathered during pre-onsite
document reviews
Review the MCP’s anticipated interview
participants, and identify topics that will
promote an inclusive discussion
If multiple reviewers are assigned to an
MCP, assign primary roles to each reviewer
(such as interviewer or note-taker), while
allowing for shared roles and
responsibilities as appropriate throughout
the onsite visit

Resources for Activity 3, Step 5
Worksheet 3.1. Compliance Review
• Includes a list of the types of documents the
MCP may provide the EQRO to demonstrate the
MCP’s compliance with federal regulations and
state standards.
• Provides space for reviewers to document followup to questions from the pre-onsite visit
documentation review. The completed
Compliance Review Worksheet is a primary data
source for analyses and a comprehensive record
of compliance protocol EQR-related activities
Worksheet 3.4. Compliance Interview Questions
• These questions are intended to guide the
reviewer's discussion with MCP staff to help
determine compliance with state and federal
requirements
• The questions are first organized by MCP staff
roles and then by regulatory provision

It is strongly recommended that the EQRO completes the onsite document review, (Activity 2,
Step 2), before the interviews. Some interview participants may provide additional documents
during their interview. This might be done when such documents are vital to the discussion or if
the review of the documents will benefit from joint review by all participants.
Interview Participants
Interviews should be conducted with groups, rather than with single individuals, because rarely
does one individual have sole responsibility for a particular function. Interview groups should
include participants that represent different functions, services, or departments of the MCP to
enable the EQRO to collect multiple perspectives about an issue. Group interviews are also an
opportunity for MCP staff to learn about compliance activities in other departments. The EQRO
has the discretion to meet with less than the full list of MCP-recommended employees in
situations where the EQRO feels that it can obtain the required information without the
attendance of all MCP employees listed in the protocol, or the MCP has identified a more
appropriate person to address questions but is not on the recommended list. Worksheet 3.4.
includes questions for the following groups:
•

MCP leaders

•

MCP information systems staff

•

Quality assessment and performance improvement program staff

•

Provider/contractor services staff

•

Enrollee services staff, including grievance and appeal staff

•

Utilization management staff

•

Medical director(s)
PROTOCOL THREE | 123

•

Case managers and care coordinators, and

•

MCP providers and contractors, as appropriate and as time and resources permit

Interview Process
The EQRO should provide the MCP interview participants with an interview agenda before the
interview, which includes the interview goals, issues, topics, and a list of related materials or
documents. Effective facilitation of an interview with an individual or a group requires that the
EQRO:
•

Maintain control of the interview discussion by politely redirecting participants to the topic or
question as necessary

•

Adhere to the time frames outlined in the agenda

•

Listen carefully to participants and summarize or restate participant responses to ensure
understanding

•

Take notes using the Worksheet 3.1 or similar tool, or according to the Compliance Review
Questions provided in Worksheet 3.4

•

Review documents provided during the interview at an appropriate time based on the
content and purpose of sharing the document

•

Conclude the interview with a review of the outlined goals and compliance levels to ensure
an understanding of the extent to which they were met, and

•

Provide information about next steps as appropriate

Interviews & Systems Capabilities
States have the opportunity to expand the roles of other state agencies in terms of their
responsibilities related to data exchanges, EHRs, interoperability, care coordination, and
Medicaid or CHIP waivers. At the state’s discretion, it may determine:
•

Whether the EQRO will review the state’s health information technology (HIT) plan for
HITECH and meaningful use with respect to validation of performance measures or
performance improvement project activities, and

•

How the MCP’s systems will support state efforts in a valid way

More information on conducting an ISCA is provide in Appendix A.
Resources to Conduct an Information Systems Capabilities Assessment (ISCA)
The ISCA is used to validate MCP information systems, processes, and data. The ISCA provides a foundation for the
validation of performance measures.
• Appendix A explains how to conduct the ISCA.
• Worksheet A.1. ISCA Tool is completed by the MCP and documents the capabilities of the information systems,
processes, and data
• Worksheet A.2. ISCA Interview Guide is used by EQROs to conduct follow-up interviews with staff to record
responses and document specific issues based on findings from Worksheet A.1

124 | PROTOCOL THREE

Step 6: Conduct Exit MCP Interviews

The EQRO conducts an exit interview at the conclusion of the onsite visit with MCP staff. The
purpose of the exit interview is to clarify the EQRO’s understanding of the information collected
throughout the compliance review process. The EQRO should provide the MCP with the
opportunity to respond to initial compliance issues to ensure the findings are due to true noncompliance and not due to misunderstanding or misinterpretation of MCP documents and
interviews.

ACTIVITY 4: COMPILE AND ANALYZE FINDINGS (POST-ONSITE VISIT)
Post site-visit activities include (1) collecting and documenting additional information as needed,
and (2) analyzing data compiled pre-, during, and post-onsite visit to make compliance
determinations for each regulatory provision.
Step 1: Collect Supplemental Information

In addition to information collected during the onsite visit, the EQRO should consider other
sources of information that confirm the MCP’s compliance with federal regulations and state
standards. Additional sources should include the following:
•

Results of Medicaid and CHIP beneficiary surveys (see Protocol 6 about administering
surveys)

•

Results of independent assessments of the MCP’s information systems (see Appendix A
about performing an ISCA)

•

Results of independent assessments of MCP encounter data (see Protocol 5 about
validating encounter data)

•

Results of independent validations of MCP performance measures (see Protocol 2 and
Protocol 7 for validating and calculating performance measures, respectively)

•

Results of independent validation of performance improvement projects (PIPs) (see
Protocol 1 and Protocol 8 about validating and implementing PIPs, respectively or Protocol
9 about conducting a Focus Study)

•

Additional materials requested during or after the onsite visit, such as grievance and appeal
reports and analyses

Step 2: Compile Data and Information

WORKSHEET 3.1

EQROs should use Worksheet 3.1 (or a similar template) to document additional information
they review, including sources of the information and their findings about the MCP’s compliance.
Step 3: Analyze Findings

WORKSHEET 3.1
WORKSHEET 3.2

PROTOCOL THREE | 125

One commonly used approach to analyzing
EQR findings is to assign a numerical value to
indicate the degree of compliance with a given
regulatory provision. The EQRO should
document both a compliance score for each
regulatory provision, as well as details and
justification for the compliance determination.

Resources for Activity 4, Step 3
Worksheet 3.1. Compliance Review
• Includes a list of the types of documents the
MCP may provide the EQRO to demonstrate the
MCP's compliance with federal regulations and
state standards
• Can be used by the EQRO to document

compliance scores and justification
Regardless of the number of points on a scale,
each level of compliance must be defined clearly
Worksheet 3.2. Compliance Definitions
for the state, the EQRO, and the MCP before
• Provides compliance definitions and examples
beginning the review. While one scale may
of compliance rating scales
serve as the primary method of assigning levels
• Compliance rating scales may be adjusted to
of compliance, it does not preclude the
best suit a state’s needs
combined use of another scale. For example, a
five-point compliance scale may be appropriate
for most of the provisions, but some provisions may be dichotomous (e.g., met or not met).
When determinations are made for levels of compliance other than ‘met’ or ‘not met,’ such as
‘partially met,’ the EQRO should clearly identify specific deficiencies, as well as the rationale for
and evidence of the deficiency.

ACTIVITY 5: REPORT RESULTS TO THE STATE
Step 1: Submit a Report Outline to the State

The EQRO should develop a report outline and submit it to the state for approval. The outline
will then be used by the EQRO to draft a report to the state with the results of the MCP’s
compliance with federal and state requirements.
Step 2: Submit a Final Determination Report to the State

WORKSHEET 3.1

Because the state may use the report to meet its
reporting requirements for federal or state agencies, the
state legislature, local advocacy groups, and other
interested parties, the state may need certain types of
information presented in a specific format. While nonsummarized findings might be of interest to some
individuals, the report should include an overall summary
of findings for compliance with regulatory provisions.

Resources for Activity 5, Step 2
Worksheet 3.1. Compliance Review
• Includes a list of the types of
documents the MCP may provide the
EQRO to demonstrate the MCP’s
compliance with federal regulations
and state standards

By design, Worksheet 3.1 separates the regulatory provisions into three major sections:
1

Standards, including enrollee rights and protections

Quality assessment and performance improvement (QAPI) program

Grievance system

126 | PROTOCOL THREE

Although each regulation is assigned a level of compliance, the EQRO and the state may group
select regulatory provisions together to combine ratings into one aggregate compliance score.
The degree of “roll-up” or aggregation will be determined by the level of reporting required by
the state to meet its reporting requirements or its interest in comparison and/or MCP incentive
programs (e.g., statewide pay for performance programs).
Step 3: Submit Other Reports Requested by the State

The state may request a specific format for reporting results back to the MCP. Some options for
reporting evaluation results to the MCP include:
1

Compliance Issues Only. Reviewers provide verbal feedback about general compliance
issues they have identified during the course of conducting the compliance review EQRrelated activity. Neither compliance determinations for individual regulatory provisions nor
findings for a level of MCP performance are discussed. This type of feedback typically is
provided to the MCP leadership during a closing session or exit interview at the onsite visit.
This provides the MCP the opportunity to offer additional information if evidence of
compliance is available

Compliance Issues Specific to Regulatory Provisions. Reviewers provide verbal
feedback for regulatory provisions or components of provisions that are determined less
than fully compliant, in accordance with the compliance thresholds established by the state
before the review. Findings for a level of MCP performance are not discussed. This type of
feedback is typically presented to the MCP leadership during a closing session or exit
interview at the onsite visit. This provides the MCP the opportunity to offer additional
information if evidence of compliance is available

Compliance Determinations and Deficiency Report. Reviewers provide verbal and/or
written feedback about identified compliance issues, compliance ratings for regulatory
provisions, and an overall finding for MCP performance, highlighting areas of deficiency that
will be presented to the state
END OF PROTOCOL 3

PROTOCOL THREE | 127

WORKSHEETS FOR PROTOCOL 3:
COMPLIANCE REVIEW TOOLS
Instructions. Use these or similar worksheets to assess the MCP’s compliance with federal regulations and state
standards. These worksheets include a tool for document review, compliance definitions, a sample site visit agenda,
and compliance review interview questions. Each worksheet can be adapted as needed. This tool includes the
following worksheets crosswalked to the applicable Activity and Step:
Worksheet name

Protocol activity and step

Worksheet 3.1. Compliance Review

Activity 1. Step 1. Collect Information from the State
Activity 2. Step 2. Perform a Document Review
Activity 3. Step 5. Conduct MCP Interviews
Activity 4. Step 2. Compile Data and Information
Activity 4. Step 3. Analyze Findings
Activity 5. Step 2. Submit a Final Determination Report

Worksheet 3.2 Compliance Definitions

Activity 1. Step 2. Define Levels of Compliance
Activity 4. Step 3. Analyze Findings

Worksheet 3.3. Sample Site Visit Agenda

Activity 3. Step 3. Develop a Site Visit Agenda

Worksheet 3.4. Compliance Review Interview
Questions

Activity 3. Step 5. Conduct MCP Interviews

128 | PROTOCOL THREE

Worksheet 3.1. Compliance Review
Instructions. Worksheet 3.1 includes a list of the types of documents the MCP may provide to the EQRO to
demonstrate the MCP’s compliance with federal regulations and state standards. It separates the regulatory
provisions into three major sections:
1. Standards, including enrollee rights and protections
2. Quality assessment and performance improvement (QAPI) program
3. Grievance system
This template may be used to track which MCP documents can provide the rationale for the compliance
determination for each regulatory provision (or component). This completed worksheet is intended to record
compliance activities to support the analyses.
Note: In the template, MCP documents are identified using generic names, except in instances where the regulatory
provisions refer to and require a specific document be present and reviewed for content.
The subject matter of each example MCP document is indicated in parenthesis as follows:
AM = Administrative/ Managerial
PS = Provider/Contractor Services
UM = Utilization Management
ES = Enrollee Services
IS = Information Systems
SP = Staff Planning, Education, Development and Evaluation
The subject matter designation does not imply that the document cannot be used as a data source for addressing
other provision issues, or that it should be the sole source of data in evaluating compliance with the provisions noted.
Refer to Worksheet 3.2, Compliance Definitions for more information on approaches to compliance scoring.

PROTOCOL THREE | 129

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Availability of
services
Medicaid: 42 C.F.R.
§§ 438.206
(availability of
services) and 42
C.F.R. § 10(h)
provider directory)
CHIP: 42 C.F.R. §
457.1230(a)

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• The state’s provider-specific
network adequacy requirements
and standards (and exceptions, if
any)
• The state’s requirements for the
MCP provider directory
• Information on the documentation
that the state uses to support its
certification that the MCP
complied with the state’s
requirements for availability and
accessibility of services, including
the adequacy of the provider
network

Applicable MCP documents
• Service planning documents and
provider network planning documents
(e.g., geographic assessments,
provider network assessments,
enrollee demographic studies,
population needs assessments)(AM)

Medicaid:

• Service availability and accessibility
expectations and standards (AM)

Not Met

• Other performance standards and
quality indicators established by the
MCP (AM)
• Any measurement or analysis reports
on service availability and accessibility
(AM)
• List of all care and service providers in
the MCP’s network (may be the same
as the provider directory) (AM)
• Organization strategic plans (AM)
• Administrative policies and procedures
(AM)
• Medicaid/CHIP and other enrollee
survey results (AM)
• Utilization management policies and
procedures (UM)
• Service authorization policies and
procedures (UM)
• Provider contracts (PS)
• Provider/Contractor procedure manuals
(PS)
• Provider/Contractor oversight and
evaluation policies and procedures,
audit tools (PS)
• Medicaid/CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid/CHIP Enrollee Handbooks
(ES)
• Medicaid/CHIP provider directory
• Medicaid/CHIP Enrollee Orientation
Curriculum (ES)
• Medicaid/CHIP enrollee grievance and
appeals policies and procedures (ES)

130 | PROTOCOL THREE

Reviewer
determination
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Furnishing of
services and timely
access
Medicaid: 42 C.F.R.
§ 438.206(c)(1):
Furnishing of
services and timely
access
CHIP: 42 CFR §
457.1230(a):
Availability of
services

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Obtain a copy of the state
Medicaid/CHIP agency’s
standards for timely enrollee
access to care and services
required of Medicaid/CHIP and
MCPs.

Applicable MCP documents

Reviewer
determination

• Service planning documents and
provider network planning documents
(e.g., geographic assessments,
provider network assessments,
enrollee demographic studies,
population needs assessments)(AM)

Medicaid:

• Service availability and accessibility
expectations and standards (AM)

Not Met

Fully Met
Substantially Met
Partially Met
Minimally Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Utilization management policies and
procedures (UM)
• Service authorization policies and
procedures (UM)
• Provider contracts (PS)
• Provider/Contractor procedure manuals
(PS)
• Provider/Contractor oversight and
evaluation policies and procedures,
audit tools (PS)
• Medicaid/CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid/CHIP Enrollee Handbooks
(ES)
• Medicaid/CHIP provider directory
• Medicaid/CHIP Enrollee Orientation
Curriculum (ES)
• Medicaid/CHIP enrollee grievance and
appeals policies and procedures (ES)

PROTOCOL THREE | 131

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Access and
cultural
considerations
Medicaid: 42 C.F.R.
§ 438.206(c)(2):
Furnishing of
services and cultural
considerations.
CHIP: 42 CFR §
457.1230(a): Access
standards

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

Reviewer
determination

• Descriptive information on the
state’s efforts to promote the
delivery of services in a culturally
competent manner to all
enrollees, including those with
limited English proficiency and
diverse cultural and ethnic
backgrounds.

• Service planning documents and
provider network planning documents
(e.g., geographic assessments,
provider network assessments,
enrollee demographic studies,
population needs assessments)(AM)

Medicaid:

Not Met

• The requirements the state has
communicated to the MCP with
respect to how the MCP is
expected to participate in the
state’s efforts to promote the
delivery of services in a culturally
competent manner.

• Service availability and accessibility
expectations and standards (AM)
• Other performance standards and
quality indicators established by the
MCP (AM)
• Any measurement or analysis reports
on service availability and accessibility
(AM)
• List of all care and service providers in
the MCP’s network (may be the same
as the provider directory) (AM)
• Organization strategic plans (AM)
• Administrative policies and procedures
(AM)
• Medicaid/CHIP and other enrollee
survey results (AM)
• Utilization management policies and
procedures (UM)
• Service authorization policies and
procedures (UM)
• Provider contracts (PS)
• Provider/Contractor procedure manuals
(PS)
• Provider/Contractor oversight and
evaluation policies and procedures,
audit tools (PS)
• Medicaid/CHIP enrollee services
policies and procedures (ES)
• Statement of enrollee rights (ES)
• Medicaid/CHIP Enrollee Handbooks
(ES)
• Medicaid/CHIP provider directory (ES)
• Medicaid/CHIP Enrollee Orientation
Curriculum (ES)
• Medicaid/CHIP enrollee grievance and
appeals policies and procedures (ES)

132 | PROTOCOL THREE

Fully Met
Substantially Met
Partially Met
Minimally Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Assurances of
adequate capacity
and services
Medicaid: 42 C.F.R.
§ 438.207:
Assurances of
adequate capacity
and services
CHIP: 42 CFR §
457.1230(b):
Assurances of
adequate capacity
and services

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Medicaid/CHIP agency
documentation and submission
timing standards to assure that
the MCP has an appropriate
range of preventive, primary care,
specialty, and LTSS services that
are adequate for the anticipated
number of enrollees in the MCP’s
service area.
• Medicaid/CHIP agency
documentation and submission
timing standards to assure that
the MCP maintains a network of
providers that is sufficient in
number, mix, and geographic
distribution to meet the needs of
the anticipated number of
enrollees in the service area.

Applicable MCP documents

Reviewer
determination

• MCP 42 C.F.R. § 438.207(b)
compliance documentation

Medicaid:

• MCP 42 C.F.R. § 438.207(c)
compliance documentation

Substantially Met

• MCP 42 C.F.R. § 457.1230(b)
compliance documentation

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

PROTOCOL THREE | 133

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance

Coordination and
continuity of care
for all enrollees

The state’s requirements regarding
the obligation to and methods by
which an MCP must:

Medicaid: 42 C.F.R.
§ 438.208:
Coordination and
continuity of care
CHIP: 42 C.F.R.
§ 457.1230(c):
Coordination and
continuity of care

• a) Ensure enrollees have an
ongoing source of care
appropriate to their needs and
a person or entity formally
designated as primarily
responsible for coordinating the
services accessed by the enrollee.
The enrollee must be provided
information on how to contact their
designated person or entity
• b) Coordinate the services
the MCP furnishes to enrollees
(between settings, between
MCPs, between MCP and FFS,
and with services provided by
community and social supports)
• c) Make a best effort to conduct
an initial screening of
each enrollee's needs, within 90
days of the effective date of
enrollment for all new enrollees
• d) Share with the state or other
MCPs serving the enrollee the
results of any identification and
assessment of that enrollee's
needs to prevent duplication of
those activities
• e) Ensure that
each provider furnishing services
to enrollees maintains and shares,
as appropriate, an enrollee health
record in accordance with
professional standards
• f) Ensure that in the process of
coordinating care, each enrollee's
privacy is protected in accordance
with applicable privacy
requirements

134 | PROTOCOL THREE

Applicable MCP documents

Reviewer
determination

• Practice guidelines adopted by the
MCP (AM)

Medicaid:

• Provider/Contractor Services policies
and procedures manuals (PS)

Substantially Met

• Provider contracts (PS)
• Provider/Contractor procedure manuals
(PS)
• Medicaid/CHIP enrollee services
policies and procedures (ES)

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable

• Medicaid/CHIP enrollment and
disenrollment policies and procedures
(ES)

Reviewer Notes:

• Medicaid/CHIP Enrollee Handbooks
(ES)

Fully Met

• Care coordination policies and
procedures, and enrollee records (ES)
• Sample of Medicaid/CHIP enrollee
records (ES)
• Medicaid/CHIP enrollment and
disenrollment policies and procedures
(ES)
• A copy of the state-MCP contract
provisions, which specify the methods
by which the MCP assures the state
Medicaid/CHIP agency that it does not
request disenrollment for reasons other
than those permitted under the
contract.

CHIP:
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Additional
coordination and
continuity of care
requirements:
LTSS
Medicaid: 42 C.F.R.
§ 438.208:
Coordination and
continuity of care
CHIP: 42 C.F.R. §
457.1230(c):
Coordination and
continuity of care

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance

Applicable MCP documents

Reviewer
determination

• Methods used by the
Medicaid/CHIP agency to identify
to the MCP enrollees who need
LTSS.

• Practice guidelines adopted by the
MCP (AM)

Medicaid:
Substantially Met

• Whether the MCP is required to
meet identification, assessment,
and treatment planning
requirements for dually-enrolled
beneficiaries.

• Provider/Contractor Services policies
and procedures manuals (PS)
• Provider contracts (PS)

• Any Medicaid/CHIP agency LTSS
assessment mechanisms
requirements, including the
requirement to use appropriate
providers or individuals meeting
the Medicaid/CHIP agency’s
LTSS service coordination
requirements.

• Provider/Contractor procedure manuals
(PS)
• Enrollee services policies and
procedures (ES)

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable

• Enrollee Handbooks (ES)

Reviewer Notes:

• Care coordination policies and
procedures, and enrollee records (ES)

CHIP:

• Sample of enrollee records (ES)

Fully Met
Substantially Met
Partially Met

• The state’s quality assurance and
utilization review standards.

Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Additional
coordination and
continuity of care
requirements:
SHCN

• Methods used by the
Medicaid/CHIP agency to identify
to the MCP individuals with
special health care needs
(SHCNs).

Medicaid: 42 C.F.R.
§ 438.208:
Coordination and
continuity of care

• Whether the MCP is required to
implement mechanisms for
identifying, assessing, and
producing a treatment plan for
persons with SHCNs using the
state’s definition of SHCNs.

CHIP: 42 C.F.R.
§ 457.1230(c):
Coordination and
continuity of care

• Whether the MCP is required to
meet identification, assessment,
and treatment planning
requirements for dually-enrolled
beneficiaries.
• Any Medicaid/CHIP agency SHCN
assessment mechanisms
requirements, including the
requirement to use appropriate
providers or individuals meeting
the Medicaid/CHIP agency’s
LTSS service coordination
requirements.

• Practice guidelines adopted by the
MCP (AM)

Medicaid:

• Provider/Contractor Services policies
and procedures manuals (PS)

Substantially Met

• Provider contracts (PS)
• Provider/Contractor procedure manuals
(PS)
• Enrollee services policies and
procedures (ES)

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable

• Enrollee Handbooks (ES)

Reviewer Notes:

• Care coordination policies and
procedures, and enrollee records (ES)

CHIP:

• Sample of enrollee records (ES)

Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Whether the Medicaid/CHIP
agency requires the MCP to
produce a treatment or service
plan for enrollees with SHCN that
are determined through
assessment to need a course of
treatment or regular care
monitoring.
• The state’s quality assurance and
utilization review standards.

PROTOCOL THREE | 135

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Disenrollment
Medicaid: 42 C.F.R.
§ 438.56:
Disenrollment:
Requirements and
limitations
CHIP: 42 C.F.R. §
457.1212:
Disenrollment

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Obtain from the Medicaid/CHIP
agency Information on:
• Reasons for which the MCP may
request the disenrollment of an
enrollee.
• Methods by which the MCP
assures the Medicaid/CHIP
agency that it does not request
disenrollment for reasons other
than those permitted under the
contract.
• Whether the state chooses to limit
disenrollment.

Applicable MCP documents
• Medicaid/CHIP enrollment and
disenrollment policies and procedures
(ES)

Reviewer
determination
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:

• Medicaid/CHIP agency enrollee
disenrollment request policies.

Fully Met

• Whether the Medicaid/CHIP
agency allows the MCP to
process enrollee requests for
disenrollment.

Partially Met

• Whether the Medicaid/CHIP
agency requires enrollees to seek
redress through the MCP’s
grievance system before the
Medicaid/CHIP agency makes a
disenrollment determination on
the enrollee’s request.

136 | PROTOCOL THREE

Substantially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Coverage and
authorization of
services
Medicaid: 42 C.F.R.
§ 438.210(a–e)*:
Coverage and
authorization of
services, including
42 C.F.R. § 440.230
Sufficiency of
amount, duration,
and scope;
42 C.F.R. § Part
441, Subpart B:
Early and Periodic
Screening,
Diagnosis, and
Treatment (EPSDT)
of Individuals Under
Age 21;* and
42 C.F.R. § 438.114,
Emergency and
post-stabilization
services
CHIP: 42 C.F.R. §
457.1230(d):
Coverage and
authorization of
services

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Obtain from the state any amount,
duration, and/or scope of service
requirements that are greater than
those set forth in 42 C.F.R. §
440.230 or, for enrollees under
the age of 21, as set forth in 42
C.F.R. § Part 441, Subpart B.
• Obtain from the state any
statutory, regulatory and policy
definitions of “medical necessity”,
as well as any quantitative and
non-quantitative treatment
limitation limits set forth in those
sources.
• Obtain from the state
Medicaid/CHIP agency the stateestablished standards for MCP
processing of standard
authorization decisions.
• Any Medicaid/CHIP agency drug
authorization requirements,
including whether the
Medicaid/CHIP agency requires
approval of outpatient drugs
before its dispensing under
Section 1927(d)(5)(A) of the Act.

Applicable MCP documents

Reviewer
determination

• Provider contracts (PS)

Medicaid:

• Contracts or written agreements with
organizational subcontractors (AM)

Fully Met

• Completed evaluations of entities
conducted before delegation is granted
(AM)

Partially Met

• Medicaid/CHIP and other enrollee
grievance and appeals data (AM)

Substantially Met
Minimally Met
Not Met
Not Applicable

• Utilization management policies and
procedures (UM)

Reviewer Notes:

• Coverage rules and payment policies
(UM)

CHIP:

• Data on claims denials (UM)

Fully Met

• Service authorization policies and
procedures (standard, expedited and
extensions) (UM)
• Policies and procedures for notifying
providers and enrollees of denials of
service (UM)

Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

42 C.F.R. §
457.1228:
Emergency and
post-stabilization
services
*Note: 42 C.F.R.
§ 438.210(a)(5),
§ 438.210(b)(2)(iii),
§ 440.230 and
§441 Subpart B do
not apply to CHIP

PROTOCOL THREE | 137

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Information
requirements for
all enrollees
Medicaid: 42 C.F.R.
§ 438.100(b)(2)(i)
Enrollee right to
receive information
in accordance with
42 C.F.R. § 438.10:
Information
requirements
CHIP: 42 C.F.R §
457.1220: Enrollee
rights
42 C.F.R §
457.1207:
Information
requirements

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Whether the Medicaid/CHIP agency,
enrollment broker, or MCP must
provide all required information to
enrollees.
• Medicaid/CHIP agency developed
definitions for managed care
terminology, including appeal, copayment, durable medical equipment,
emergency medical condition,
emergency medical transportation,
emergency room care, emergency
services, excluded services, grievance,
habilitation services and devices,
health insurance, home health care,
hospice services, hospitalization,
hospital outpatient care, medically
necessary, network, non-participating
provider, physician services, plan,
preauthorization, participating provider,
premium, prescription drug coverage,
prescription drugs, primary care
physician, primary care provider,
provider, rehabilitation services and
devices, skilled nursing care, specialist,
and urgent care.
• Medicaid/CHIP agency developed
model enrollee handbooks and
enrollee notices.
• The language(s) that the
Medicaid/CHIP agency determines are
prevalent in the MCP’s geographic
service area, and all non-English
languages that the Medicaid/CHIP
identifies.
• Policies relevant to written material
language and format, for example,
policies relevant to inclusion of taglines.
• Any interpretation services that the
Medicaid/CHIP agency makes
available to enrollees.
• How the Medicaid/CHIP agency
defines ‘reasonable time’ for purposes
of providing the enrollee handbook to
enrollees.
• Medicaid/CHIP agency developed or
approved language describing
grievance, appeal, and fair hearing
procedures and timeframes, for
inclusion in the enrollee handbook.
• Medicaid/CHIP agency policy on
whether enrollee are required to pay
costs for services while an appeal or
state fair hear is pending – and the final
decision is adverse to the enrollee – for
purposes of the enrollee handbook.
• Any content required by the state for
the enrollee handbook that is not
covered in 42 CFR 438.10(g).
• Information on how the state has
defined a “significant change” in the
information MCPs are required to give
enrollees pursuant to 42 C.F.R. §
438.10(g).
• Any applicable Medicaid/CHIP laws on
enrollee rights.

138 | PROTOCOL THREE

Applicable MCP documents

Reviewer
determination

• Medicaid/CHIP and other enrollee
survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met

Fully Met

• Enrollee services policies and
procedures (ES)

Partially Met

• Statement of enrollee rights (ES)

Not Met

• Enrollee marketing materials
• Medicaid/CHIP marketing plans,
policies and procedures (ES)
• Medicaid/CHIP enrollment and
disenrollment policies and procedures
(ES)

Minimally Met
Not Applicable
Reviewer Notes:
CHIP:

• Enrollee Handbooks (ES)

Fully Met

• Enrollee grievance and appeals
policies and procedures (ES)

Substantially Met
Partially Met

• Staff Handbooks (SP)

Minimally Met

• Staff Orientation and Training
Curriculum (SP)

Not Met

• MCP provider directory (ES)
• MCP Formulary (ES)
• MCP website (ES)

Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Enrollee right to
receive information
on available
treatment options
Medicaid: 42 C.F.R.
§ 438.100(b)(2)(iii)
Enrollee right to
receive information
on available
treatment options
and alternatives . . .
including
requirements of 42
C.F.R. § 38.102:
Provider-enrollee
communications
CHIP: 42 C.F.R. §
457.1222: Providerenrollee
communication

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Information on whether or not the
MCP has documented to the state
any moral or religious objection to
providing, reimbursing for, or
providing coverage of, a
counseling or referral service for a
particular Medicaid/CHIP service
or services.

Applicable MCP documents
• Medicaid/CHIP and other enrollee
survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met
Partially Met

• Statement of enrollee rights (ES)

Not Met

• Medicaid/CHIP enrollee marketing
materials (ES)

Medicaid: 42 C.F.R.
§ 438.100(b)(2)(iv)
and (v): Enrollee
right to:
- participate in
decisions regarding
his or her care,
including the right to
refuse treatment;
- Be free from any
form of restraint . . .
as specified in other
Federal regulations
And related:
42 C.F.R. § 438.3(j):
Advance directives
CHIP: 42 C.F.R. §
457.1220: Enrollee
rights

• Information on whether or not the
MCP has documented to the state
any moral or religious objection to
fulfilling the regulatory provisions
pertaining to advance directives

Minimally Met
Not Applicable

• Medicaid/CHIP marketing plans,
policies and procedures (ES)

Reviewer Notes:

• Medicaid/CHIP enrollment and
disenrollment policies and procedures
(ES)

CHIP:

• Medicaid/CHIP Enrollee Handbooks
(ES)

Substantially Met

• Medicaid/CHIP Enrollee Orientation
Curriculum (ES)

Minimally Met

• Medicaid/CHIP enrollee grievance and
appeals policies and procedures (ES)

• A written description of any state
law(s) concerning advance
directives. The written description
may include information from state
statutes on advance directives,
regulations that implement the
statutory provisions, opinions
rendered by state courts and other
states administrative directives.
[Note to reviewers: Each state
Medicaid/CHIP agency is required
under Federal regulations at 42
C.F.R. § 431.20 to develop such a
description of state laws and to
distribute it to all MCPs. Revisions
to this description as a result of
changes in State law are to be
sent to MCPs no later than 60
days from the effective date of the
change in state law.]

Fully Met

• Medicaid/CHIP enrollee services
policies and procedures (ES)

• Staff Handbooks (SP)

Enrollee right to
participate in
decisions
regarding his or
her care and be
free from any form
of restraint

Reviewer
determination

Fully Met
Partially Met
Not Met
Not Applicable

• Staff Orientation and Training
Curriculum (SP)

Reviewer Notes:

• Medicaid/CHIP and other enrollee
survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met

Fully Met

• Medicaid/CHIP enrollee services
policies and procedures (ES)

Partially Met

• Statement of enrollee rights (ES)

Not Met

• Medicaid/CHIP enrollee marketing
materials (ES)

Minimally Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

PROTOCOL THREE | 139

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Compliance with
other Federal and
state laws
Medicaid: 42 C.F.R.
§ 438.100(d):
Compliance with
other federal and
state laws

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Obtain from the state
Medicaid/CHIP agency the
identification of all State laws that
pertain to enrollee rights and with
which the state Medicaid/CHIP
Agency requires its MCPs to
comply.

Applicable MCP documents
• Medicaid/CHIP and other enrollee
survey results (AM)

Medicaid:

• Provider contracts (PS)

Substantially Met
Partially Met

• Statement of enrollee rights (ES)

Not Met

CHIP: 42 C.F.R. §
457.1233(a):
Provider selection

• Obtain from the state information
on any credentialing, recredentialing, or other provider
selection and retention
requirements established by the
state that address acute, primary,
behavioral, substance use
disorder, and MLTSS providers,
as appropriate.

• Medicaid/CHIP enrollment and
disenrollment policies and procedures
(ES)

CHIP:

• Medicaid/CHIP Enrollee Handbooks
(ES)

Substantially Met

• Medicaid/CHIP Enrollee Orientation
Curriculum (ES)

Minimally Met

Fully Met
Partially Met
Not Met
Not Applicable

• Staff Orientation and Training
Curriculum (SP)

Reviewer Notes:

• Service planning documents and
provider network planning documents
(e.g., geographic assessments,
provider network assessments,
enrollee demographic studies,
population needs assessments) (AM)

Medicaid:

• Contracts or written agreements with
organizational subcontractors (AM)

Not Met

• Procedures and methodology for
oversight, monitoring, and review of
delegated activities (AM)
• Contracts or written agreements with
organizational subcontractors (AM)
• Completed evaluations of entities
conducted before delegation is granted
(AM)
• Provider/Contractor files, 15-20
individual health care professional files,
and 15-20 institutional provider files
(PS)
• Credentialing committee or other
provider review mechanism meeting
minutes (PS)
• Sample of files of practitioners who
have not been appointed or
reappointed (PS)

140 | PROTOCOL THREE

Not Applicable
Reviewer Notes:

• Staff Handbooks (SP)

Medicaid: 42 C.F.R.
§ 438.214: Provider
selection

Minimally Met

• Medicaid/CHIP marketing plans,
policies and procedures (ES)

• Medicaid/CHIP enrollee grievance and
appeals policies and procedures (ES)

Provider Selection

Fully Met

• Medicaid/CHIP enrollee services
policies and procedures (ES)
• Medicaid/CHIP enrollee marketing
materials (ES)

CHIP: 42 C.F.R. §
457.1220: Enrollee
rights

Reviewer
determination

Fully Met
Substantially Met
Partially Met
Minimally Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Sub-contractual
relationships and
delegation
Medicaid: 42 C.F.R.
§ 438.230:
Subcontractual
relationships and
delegation

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Obtain from the state the “periodic
schedule” established by the State
according to which the MCP is to
monitor and formally review on an
ongoing basis all subcontractors’
performance of any delegated
activities.

CHIP: 42 C.F.R. §
457.1233(b):
Subcontractual
relationships and
delegation

Applicable MCP documents
• Procedures and methodology for
oversight, monitoring, and review of
delegated activities (AM)
• Contracts or written agreements with
organizational subcontractors (AM)
• Completed evaluations of entities
conducted before delegation is granted
(AM)
• Ongoing evaluations of entities
performing delegated activities

Reviewer
determination
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Practice Guidelines
Medicaid: 42 C.F.R.
§ 438.236: Practice
guidelines
CHIP: 42 C.F.R. §
457.1233(c):
Practice guidelines

• Information on any state statutory,
regulatory, or policy requirements
concerning MCP practice
guidelines.

• Provider contracts (PS)

Medicaid:

• Contracts or written agreements with
organizational subcontractors (AM)

Fully Met

• Practice guidelines (AM)

Partially Met

• Provider/Contractor Services policies
and procedures manuals (PS)

Minimally Met

• Medicaid/CHIP enrollee services
policies and procedures (ES)

Not Applicable

Substantially Met

Not Met

Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

PROTOCOL THREE | 141

MCP Standards, Including Enrollee Rights and Protections
Federal regulation
source(s)
Health information
systems
Medicaid: 42 C.F.R.
§ 438.242
CHIP: 42 C.F.R. §
457.1233(d):

Medicaid/CHIP agency policy/
regulation information needed to
determine MCP compliance
• Information on whether or not the
state has required the MCP to
undergo, or has otherwise
received, a recent assessment of
the MCP’s health information
system. If the state has required
or received such an assessment,
obtain a copy of the information
system assessment from the state
or the MCP. Also obtain contact
information about the person or
entity that conducted the
assessment and to whom followup questions may be addressed.
• State specifications for data on
enrollee and provider
characteristics that must be
collected by the MCP.
• Information on whether or not the
state has conducted a recent
review and validation of the
MCP’s encounter data, or required
the MCP to undergo, or has
otherwise received, a recent
validation of the MCP’s encounter
data. If the state has required or
received such a validation review,
obtain a copy of the review from
the state or the MCP. Also obtain
contact information about the
person or entity that conducted
the validation and to whom followup questions may be addressed.
• State specifications for how MCPs
are to (1) collect data elements
necessary to enable the
mechanized claims processing
retrieval systems to provide for
electronic transmission of claims
data in the format consistent with
the Transformed Medicaid
Statistical Information System (TMSIS); (2) collect and transmit
data on enrollee and provider
characteristics specified by the
state, on all services furnished to
enrollees through an encounter
data system; and (3) Ensure that
data received from providers is
accurate and complete.
• Specifications for submitting
encounter data to the
Medicaid/CHIP agency in
standardized ASC X12N 837 and
NCPDP formats, and the ASC
X12N 835 format.
• Make all collected data available
to the state and upon request to
CMS.
• The state’s procedures and quality
assurance protocols to ensure
that enrollee encounter data
submitted by the MCP is a
complete and accurate
representation of the services
provided to its enrollees.

142 | PROTOCOL THREE

Applicable MCP documents
• QAPI project descriptions, including
data sources and data audit results
(AM)
• Medicaid/CHIP and other enrollee
grievance and appeals data (AM)
• Analytic reports of service utilization
(UM)

Reviewer
determination
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met

• Information systems capability
assessment reports (IS)

Not Applicable

• Policies and procedures for auditing
data or descriptions of other
mechanisms used to check the
accuracy and completeness of data
(internally generated and externally
generated data) information system

Reviewer Notes:

• Completed audits of data or other
evidence of data monitoring for
accuracy and completeness both for
MCP data and information system

Partially Met

CHIP:
Fully Met
Substantially Met
Minimally Met
Not Met

• Provider/Contractor Services policies
and procedures manuals (PS)

Not Applicable

• Provider contracts (PS)

Reviewer Notes:

Quality Assessment and Performance Improvement Program
Federal regulation
source(s)
Quality
Assessment and
Performance
Improvement:
General rules
Medicaid: 42 C.F.R.
§ 438.330(a):
General rules
CHIP: 42 C.F.R. §
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation information
needed to determine MCP
compliance
• In the event that CMS specifies
national performance measures
or PIP topics, whether or not the
state has requested an exemption
from the national performance
measures or PIPs.

Applicable MCP documents
• MCP QAPI implementation
documentation (AM)

PROTOCOL THREE | 143

Quality Assessment and Performance Improvement Program
Federal regulation
source(s)
Basic elements of
quality
assessment and
performance
improvement
program
Medicaid: 42 C.F.R.
§ 438.330(b): Basic
elements of quality
assessment and
performance
improvement
programs
CHIP: 42 C.F.R. §
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation information
needed to determine MCP
compliance
• The state’s specifications for
performance improvement
projects (PIPs) required per
paragraph (d) of this section.
• The state’s specifications for how
the MCP should identify, measure
and report performance measures
required per paragraph (c) of this
section.
• The state’s requirements for
detection by the MCP of over- and
under-utilization.
• The state’s requirements for
assessment by the MCP of the
quality and appropriateness of
care furnished to enrollees with
special health care needs, as
defined in the state’s quality
strategy under 438.340 (as crossreferenced for CHIP in
457.1240(e)).
• The state’s requirements for
assessment by the MCP of the
quality and appropriateness of
care furnished using LTSS, if
applicable, including assessment
of care between care settings and
a comparison of services and
supports received with those set
forth in the enrollee’s
treatment/service plan.
• The state’s requirements for the
MCP’s participation in efforts by
the State to prevent, detect,
report, investigate and remediate
critical incidents, that occur within
the delivery of LTSS as well as to
track and trend results in order to
make systems improvements, if
applicable

144 | PROTOCOL THREE

Applicable MCP documents

Reviewer
determination

• Policies and procedures related to
QAPI project metrics (AM)

Medicaid:

• QAPI project quality indicators, the
selection or development criteria, and
processes for selection or development
(AM)

Substantially Met

Fully Met
Partially Met
Minimally Met

• Performance standards and quality
indicators established by the MCP
(AM)

Not Met

• Performance measure reports and data
provided to the state (AM)

Reviewer Notes:

• Utilization management policies and
procedures (UM)

CHIP:

• Medicaid/CHIP and other enrollee
MLTSS tracking reports (AM)

Fully Met
Substantially Met

• Policies and procedures related to data
collection and data quality checks for
QAPI projects (AM)

Partially Met

• Policies and procedures for
assessment of MLTSS services
between care settings and comparison
of services and supports received with
those set forth in the enrollee's
treatment/service plan (AM)
• Policies and procedures for assisting
the state in the prevention, detection
and remediation of critical incidents
that occur within the delivery of
MMLTSS.

Not Applicable

Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Quality Assessment and Performance Improvement Program
Federal regulation
source(s)
Performance
measurement
Medicaid: 42 C.F.R.
§ 438.330(c):
Performance
measurement
CHIP: 42 C.F.R. §
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation information
needed to determine MCP
compliance
• Information on the standard
performance measures identified
by the state.

Applicable MCP documents
• Performance measure reports and data
provided to the state (AM)

Reviewer
determination
Medicaid:
Fully Met
Substantially Met

• For an MCP providing long-term
services and supports, the
standard performance measures
relating to quality of life,
rebalancing, and community
integration activities
for individuals receiving long-term
services and supports.

Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• Information on whether the MCP
calculates the performance
measure and reports to the state
or whether the MCP provides data
to the state, which then calculates
the PM.

CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Performance
improvement
projects

• Information on any PIP
requirements specified by the
state.

Medicaid: 42 C.F.R.
§ 438.330(d) and
CHIP: 42 C.F.R. §
457.1240(b)

• Information on how often the state
requests that each MCP report
the status and results of each
project conducted per paragraph
(d)(1) of this section.
• Information on if the state permits
an MCP exclusively serving dual
eligibles to substitute an MA
Organization quality improvement
project conducted under §
422.152(d) of this chapter for one
or more of the performance
improvement projects otherwise
required under this section.

• Reports and status documentation of
MCP internal QAPI evaluations (AM)

Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

PROTOCOL THREE | 145

Quality Assessment and Performance Improvement Program
Federal regulation
source(s)
QAPI evaluations
review
Medicaid: 42 C.F.R.
§ 438.330(e)(2):
Program and review
by the state
CHIP: 42 C.F.R. §
457.1240(b): Quality
assessment and
performance
improvement
program

State policy/regulation information
needed to determine MCP
compliance
• Information on whether the state
requires its MCPs to develop a
process to evaluate the impact
and effectiveness of its
own quality assessment and
performance improvement
program. If so, information on the
frequency with which that
evaluation must be conducted,
and on the state’s requirements
for how MCPs conduct that
process.

Applicable MCP documents
• Reports and status documentation of
MCP internal QAPI evaluations (AM)

146 | PROTOCOL THREE

Grievance System
Federal Regulation
Source(s)

State Policy/Regulation
Information Needed to Determine
MCP Compliance

Grievance Systems

• Obtain information on:

Medicaid: 42 C.F.R.
§ 438.228:
Grievance and
appeal systems

• Whether or not the Medicaid/CHIP
agency delegates responsibility to
the MCP for providing each
enrollee (who has received an
adverse decision with respect to a
request for a covered service)
notice that he or she has the right
to a state fair hearing or review to
reconsider their request for the
covered service.

Applicable MCP Documents

Reviewer
Determination

• Enrollee grievance and appeals
policies and procedures (ES)

Medicaid:

• Enrollee grievance and appeal
tracking reports (ES)

Substantially Met

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

General
requirements
Medicaid: 42 C.F.R.
§ 438.402: General
requirements
CHIP: 42 C.F.R. §
457.1260: Grievance
system

• Information on:
• Whether enrollees are required or
permitted to file a grievance with
either the state or the MCP, or
both.
• Whether providers, or authorized
representatives, can act on behalf
of the enrollee to request an
appeal, file a grievance, or request
a state fair hearing or review
request.
• Whether state offers external
medical review.

• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)
• Medicaid/CHIP and other enrollee
grievance and appeals data (AM)
• Analytic reports of service utilization
(UM)
• Information systems capability
assessment reports (information
systems)
• Policies and procedures for auditing
data or descriptions of other
mechanisms used to check the
accuracy and completeness of both
internally generated and externally
generated data (Information
systems)
• Completed audits of data or other
evidence of data monitoring for
accuracy and completeness both for
MCP data and contractor (delegate)
data (information systems)
• Provider/Contractor Services policies
and procedures manuals (PS)

• Provider contracts (PS)

PROTOCOL THREE | 147

Grievance System
Federal Regulation
Source(s)
Timely and
Adequate Notice of
Adverse Benefit
Determination
Medicaid: 42 C.F.R.
§ 438.404: Timely
and adequate notice
of adverse benefit
determination
CHIP: 42 C.F.R. §
457.1260: Grievance
system

State Policy/Regulation
Information Needed to Determine
MCP Compliance
• Information on the timeframes
within which it requires MCPs to
make standard (initial) coverage
and authorization decisions and
provide written notice to requesting
enrollees. These timeframes will be
the required period within which
MCPs must provide Medicaid/CHIP
enrollees written notice of any
intent to deny or limit a service (for
which previous authorization has
not been given by the MCP) and
the enrollee’s right to file an MCP
appeal.

Applicable MCP Documents

Reviewer
Determination

• Data on claims denials (UM)

Medicaid:

• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)

Fully Met

• MCP adverse benefit determinations
(ES)
• Timing data on adverse benefit
determination mailings (ES)

Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Handling of
Grievances and
Appeals
Medicaid: 42 C.F.R.
§ 438.406: Handling
of grievances and
appeals
CHIP: 42 C.F.R. §
457.1260: Grievance
system

• Information on any state
requirements concerning handling
of grievances and appeals that
differ from those required under
438.406.
• *Note: See the ‘Disenrollment’
section in Worksheet 3.2 above for
grievances during disenrollment.

• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)
• Medicaid/CHIP and other enrollee
grievance and appeals data (AM)

148 | PROTOCOL THREE

Grievance System
Federal Regulation
Source(s)
Resolution and
notification:
Grievances and
appeals
Medicaid: 42 C.F.R.
§438.408:
Resolution and
notification,
Grievances and
appeals
CHIP: 42 C.F.R. §
457.1260: Grievance
system

State Policy/Regulation
Information Needed to Determine
MCP Compliance
• Information on:
• The state-established standard
time frames during which the state
requires MCPs to (1) dispose of a
grievance and notify the affected
parties of the result, and (2) resolve
appeals and notify affected parties
of the decision.

Applicable MCP Documents
• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)
• Medicaid/CHIP enrollee grievance
and appeal tracking reports (ES)
• MCP appeal resolution notices (ES)

Reviewer
Determination
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable

• The methods prescribed by the
state that the MCP must follow to
notify an enrollee of the disposition
of a grievance.

Reviewer Notes:

• Information on whether providers,
or authorized representatives, can
act on behalf of the enrollee to
request an appeal, file a grievance,
or request a state fair hearing
request.

CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Expedited
resolution of
appeals

• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)

Medicaid: 42 C.F.R.
§ 438.410:
Expedited resolution
of appeals

• Medicaid/CHIP enrollee grievance
and appeal tracking reports (ES)

CHIP: 42 C.F.R. §
457.1260: Grievance
system

PROTOCOL THREE | 149

Grievance System
Federal Regulation
Source(s)
Information about
the grievance
system to
providers and
subcontractors
Medicaid: 42 C.F.R.
§ 438.414:
Information about
the grievance and
appeal system to
providers and
subcontractors
CHIP: 42 C.F.R. §
457.1260: Grievance
system

State Policy/Regulation
Information Needed to Determine
MCP Compliance
• Information on:
• Whether the state develops or
approves the MCP’s description of
its grievance system that the MCP
is required to provide to all
Medicaid/CHIP enrollees (per
438.10(g)(2)(xi). [Note that under
regulations at 42 C.F.R. §
438.10(g)(1) the state must either
develop a description for use by the
MCP or approve a description
developed by the MCP.]

Applicable MCP Documents

Reviewer
Determination

• Contracts or written agreements with
organizational subcontractors (AM)

Medicaid:

• Completed evaluations of entities
conducted before delegation is
granted (AM)

Substantially Met

• Provider contracts (PS)
• Provider/Contractor procedure
manuals (PS)

Fully Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

• If the states approves, rather than
develops, the description of the
MCP’s grievance system,
information on whether or not the
state has already approved the
MCP’s description.

CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Recordkeeping
requirements
Medicaid: 42 C.F.R.
§ 438.416:
Recordkeeping
requirements
CHIP: 42 C.F.R. §
457.1260: Grievance
system

• Information on any audits or other
reviews of MCP records of
grievances and appeals conducted
by the state.

• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)
• Medicaid/CHIP enrollee grievance
and appeal tracking reports (ES)
• Sample records of grievances and
appeals (ES)

Continuation of
benefits while the
MCP appeal and
the state Fair
Hearing are
pending

• Information on any state
requirements concerning
continuation of benefits pending
appeal and state fair hearing that
differ from those required under 42
C.F.R. § 420.

42 C.F.R. § 438.420:
Continuation of
benefits while the
MCO, PIHP, or
PAHP appeal and
the state fair hearing
are pending

• Information on any audits or other
reviews of MCP records of appeals
conducted by the state, to
determine MCP compliance with
federal continuation of benefits
requirements.

(Note: This
requirement does
not apply to CHIP)

• Whether state permits managed
care plans to recover the cost of
services. See (d) reference to
“state’s usual policy.”

150 | PROTOCOL THREE

• Medicaid enrollee grievance and
appeals policies and procedures
(ES)

Medicaid-only:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes:

Grievance System
State Policy/Regulation
Information Needed to Determine
MCP Compliance

Federal Regulation
Source(s)
Effectuation of
reversed appeal
resolutions
Medicaid: 42 C.F.R.
§ 438.424:
Effectuation of
reversed appeal
resolutions.

• Information on which entity- the
state or the MCP- is required to pay
for services when the state fair
hearing officer reversed a decision
to deny authorization of services,
and the enrollee received the
disputed services while the appeal
was pending.

Applicable MCP Documents
• Medicaid/CHIP enrollee grievance
and appeals policies and procedures
(ES)

Reviewer
Determination
Medicaid:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable

CHIP: 42 C.F.R. §
457.1260: Grievance
system

Reviewer Notes:
CHIP:
Fully Met
Substantially Met
Partially Met
Minimally Met
Not Met
Not Applicable
Reviewer Notes::

Note: MCP documents are identified using generic names, except in instances where the regulatory provisions refer
to and require a specific document be present and reviewed for content.
The subject matter of each example MCP document is indicated in parenthesis as follows:
AM = Administrative/ Managerial
PS = Provider/Contractor Services
UM = Utilization Management
ES = Enrollee Services
IS = Information Systems
SP = Staff Planning, Education, Development and Evaluation
The subject matter designation does not imply that the document cannot be used as a data source for addressing
other provision issues, or that it should be the sole source of data in evaluating compliance with the provisions noted.

PROTOCOL THREE | 151

Worksheet 3.2. Compliance Definitions
Instructions. This worksheet provides examples of compliance scoring and compliance definitions. Either method
can be used by a state to distinguish levels of MCP compliance.
1. Compliance Scoring. One commonly used approach to analyzing compliance review findings is to assign a
numerical value to indicate the degree of compliance with a given regulatory provision. An EQRO can provide a
compliance score for each regulatory provision on Worksheet 3.1. Compliance Review, followed by details and
justification for the compliance determination. Before the review, the state is directed to define what constitutes
compliance and determine the rating or scoring system for what the EQRO will review. The state and EQRO can
adapt a compliance rating scale to best suit their needs. Some examples include:
 Two-point rating or scoring. Either the requirement is met or not met:
•

Met = 1

•

Not Met = 2

 Three-point rating or scoring. This scale provides credit when a requirement is partially met:
•

Fully Met = 1

•

Partially Met = 2

•

Not Met = 3

 Five-point rating or scoring. This scale allows for the scoring of all five levels of compliance:
•

Fully Met = 1

•

Substantially Met = 2

•

Partially Met = 3

•

Minimally Met = 4

•

Not Met = 5

One of the above rating or scoring scales may serve as the primary system, or alternative scales may be adapted to
certain regulatory provisions. In an extensive compliance review, the state may assert that the definition of
compliance for most regulatory provisions are appropriate for a 5-point rating scale, with two or three particular
provisions rated as “met” or “unmet.”
2. Compliance Definitions Options. The following definitions describe the extent of compliance with a given
regulatory provision:
 Full compliance:
•

All documentation listed under a regulatory provision, or component thereof, is present, and

•

MCP staff provide responses to the EQRO that are consistent with each other and with the documentation;
or

•

A state-defined percentage of all data sources—either documents or MCP staff—provide evidence of
compliance with regulatory provisions

 Substantial Compliance:
•

After review of the documentation and discussion with MCP staff, it is determined that the MCP has met
most of the requirements as stated above

 Partial Compliance:

152 | PROTOCOL THREE

•

All documentation listed under a regulatory provision, or component thereof, is present, but MCP staff are
unable to consistently articulate evidence of compliance; or

•

MCP staff can describe and verify the existence of compliant practices during the interview(s), but required
documentation is incomplete or inconsistent with practice; or

•

Any combination of “Met,” “Partially Met” and “Not Met” determinations for smaller components of a
regulatory provision would result in a “Partially Met” designation for the provision as a whole

 Minimal Compliance:
•

After review of the documentation and discussion with MCP staff, it is determined that although some
requirements have been met, the MCP has not met most of the requirements

 Non-compliance:
•

No documentation is present and MCP staff have little to no knowledge of processes or issues that comply
with regulatory provisions; or

•

No documentation is present and MCP staff have little to no knowledge of processes or issues that comply
with key components (as identified by the state) of a multi-component regulatory provision, regardless of
compliance determinations for remaining, non-key components of the regulatory provision

About Targeted Regulatory Components
If all applicable federal requirements are met, the state may focus on specific aspects or components of its regulatory
provisions to make performance improvement more manageable and targeted. If less than full compliance with a full
set of state regulations is defined by the state as acceptable, the state must identify to the EQRO and the MCP
specific regulatory provisions of the compliance review for which the MCP is accountable. This must take place
before the review begins. However, over the three-year compliance review cycle, the EQRO must review all
compliance requirements.

PROTOCOL THREE | 153

Worksheet 3.3. Sample Onsite Visit Agenda
Instructions. This worksheet provides a template to develop the onsite visit agenda. The agenda is intended to help
both the MCP and EQRO in planning staff interviews, gathering documentation, and determining logistics (such as
meeting space).
The EQRO should prepare the agenda and send it in advance to the individual representing the MCP in the
regulatory compliance review process. The MCP representative is responsible for identifying additional MCP
participants. The agenda should also provide the locations where the meetings and any additional document review
will occur. The EQRO should determine the number of days for the onsite visit based on the estimated duration and
number of meetings required to carry out the onsite visit. At the end of each day, the EQRO should lead a concluding
meeting to discuss outstanding information and answer questions. At the end of the final day, the EQRO should
provide concluding remarks and identify next steps in the review process.
Sample EQRO Onsite Visit Agenda (Add more days as needed)
•

Introductions between the EQRO reviewers and MCP participants

•

Onsite Visit Purpose: To clarify MCP compliance with federal Medicaid managed care regulations

•

Day 1 Activities: [Insert number of interviews and document review, if applicable]

•

Day 1 Final Meeting [Insert time]: Identification/discussion of outstanding issues
Time

Regulatory
compliance issue

Location

Participants

Comments/Documents

(EQRO)

(MCP)

(BOTH)

(EQRO)

(MCP)

(BOTH)

(EQRO)

Add more rows as
needed
Note: The organization responsible for completing each part of the agenda is identified above in parenthesis.
•

Day 2 Activities: [Insert number of interviews and document review, if applicable]

•

Day 2 Final Meeting [Insert time]: Concluding issues and comments; description of next steps

•

Day 2 Exit Interviews [Insert time]: Opportunity to respond to initial compliance issues (if applicable) and clarify
reviewer understanding
Time

Regulatory
compliance issue

Location

Participants

Comments/Documents

(EQRO)

(MCP)

(BOTH)

(EQRO)

(MCP)

(BOTH)

(EQRO)

Add more rows as
needed
*The organization responsible for completing each part of the agenda is identified above in parenthesis.

154 | PROTOCOL THREE

Worksheet 3.4. Compliance Interview Questions
Instructions. Worksheet 3.4 includes a list of questions the reviewer may ask MCP staff to help determine
compliance with state and federal requirements as summarized in Activity 3, Step 6. The purpose of the MCP
interviews is to collect data to supplement and verify what is learned through the preliminary document review and
onsite document review. The questions are first organized by MCP staff roles, and then broken out by regulatory
provision.
Reviewers are encouraged to interview MCP staff in appropriate groups whenever possible in order to accomplish a
comprehensive review from more than one perspective, and to achieve efficient and productive interviews.
The MCP interviewee groups who are most often interviewed are included in this guide:
•

MCP leaders

•

MCP information systems staff

•

Quality assessment and performance improvement program staff

•

Provider/contractor services staff

•

Enrollee services staff

•

Utilization management staff

•

Medical directors

•

Case managers and care coordinators

•

MCP providers and contractors (as appropriate)

The EQRO should advise the MCP of the specific issues for which the MCP will be interviewed during the site visit.
The MCP representative for the compliance review process should select and report to the EQRO in writing the
membership of each of the interviewee groups that are capable of responding to the EQRO site visit interview topic
requests.

PROTOCOL THREE | 155

MCP Leaders Interview
The leadership interview is an opportunity to speak with the senior representatives of the MCP about their
understanding of MCP requirements. MCP leaders include:
•

Chief executive officer (CEO)

•

Chief operating officer (COO)

•

Chairman of the governing body, or a representative

•

Medical director (including psychiatric medical director, if applicable)

•

Chief elected or appointed officer of the MCP 's licensed independent practitioners

•

Chief information officer (CIO)

•

Compliance officer

•

Quality improvement committee chairperson

•

Quality improvement program director or coordinator, and

•

Human resources leader

As determined by the MCP representative, usually in consultation with the CEO, other senior staff of the MCP may
also be in attendance. However, attendance at this interview should be carefully limited in order to foster candor and
exchange of information.
I. MCP Standards and Enrollee Rights and Protections
Availability of services (42 C.F.R. §§ 438.206 and 457.1230(a))
1.

Please describe the MCP’s process for assessing whether its network of appropriate providers 60 is sufficient
to provide adequate access to each type of covered service and major specialty within each type of covered
service.
a.

What issues were considered in the assessment process?

How does the MCP determine the adequacy of its network to serve its Medicaid and CHIP enrollees?

What assumptions and methodologies are used to project the number, type (training, experience, and
specialization), and location of primary care providers and specialists necessary to serve Medicaid and
CHIP enrollees?

What assumptions and methodologies are used to project the number, type (training, experience, and
specialization), and location of LTSS providers necessary to serve Medicaid enrollees?

If the state has established access requirements for LTSS, how does the MCP evaluate its current network
in comparison to the requirements?
a.

Are there any areas where the requirements are not met? If so, how is the MCP remedying these gaps?

Per 42 C.F.R. §§ 438.2 and 457.10, provider means any individual or entity that is engaged in the delivery of services, or
ordering or referring for those services, and is legally authorized to do so by the state in which it delivers the services.

156 | PROTOCOL THREE

Out-of-network providers (42 C.F.R. §§ 438.206(b)(3) through (5) and 457.1230(a))
1.

Approximately what proportion of Medicaid and CHIP enrollee provider encounters are made to out-ofnetwork providers? If this is a significant percent, what are the reasons for this?

What are the reimbursement methods for out-of-network providers? Which types of providers are paid using
each method?
[Probe: Do you receive claim, encounter data from out-of- network providers similar to the claim, or
encounter data that you receive from your network providers?]

How does your MCP ensure that any costs to the Medicaid and CHIP enrollee for out-of-network services is
no greater than the costs the enrollee would incur if they used a network provider for the same service?

Furnishing of services and timely access (42 C.F.R. §§ 438.206(c)(1) and 457.1230(a))
1.

Please describe how the MCP monitors compliance with its Medicaid and CHIP standards for timely access
to care and services.

How does the MCP ensure the 24 hours per day, 7 day per week availability of Medicaid and CHIP services
included in its contract with the state when medically necessary?

How does the MCP determine that the individual and institutional providers it contracts with have sufficient
capacity to make services available when medically appropriate 24 hour per day, 7 days per week to
Medicaid and CHIP enrollees?

How does the MCP ensure that its provider network’s hours of operation do not discriminate against
Medicaid and CHIP enrollees (i.e., are not different for Medicaid and CHIP enrollees than for commercial
enrollees)?

How is inappropriate use of emergency department visits addressed? What proportion of emergency
department visits are potentially avoidable?

What was the volume of denied claims for emergency and post-stabilization services in the most recent
year?

Access and cultural considerations (42 C.F.R. §§ 438.206(c)(2) and 457.1230(a))
1.

What have been the state Medicaid and CHIP agency’s efforts to promote the delivery of services in a
culturally appropriate manner to all enrollees, including those with diverse cultural and ethnic backgrounds?
a. How has your MCP participated in these efforts?
b. What documentation exists describing your efforts and the results of these efforts?

What efforts has the MCP made to promote services to enrollees with limited English proficiency and those
with low literacy?

How does the MCP maintain and make available information on all languages (including both spoken and
signed) used by providers, including those used by LTSS providers?

How are call center staff made aware of MCP beneficiaries’ needs so that verbal communication is easily
understood by the beneficiary? For example, volume or speed of speech.

Assurances of adequate capacity and services (42 CFR §§ 438.207(b) – (c) and 457.1230(b))

PROTOCOL THREE | 157

Please describe how your MCP demonstrates to the Medicaid and CHIP agency that it offers an appropriate
range of preventive, primary care, specialty services, and LTSS that is adequate for the anticipated number
of enrollees in your service area.

Please describe how your MCP demonstrates to the Medicaid and CHIP agency that it maintains a network
of providers that is sufficient in number, mix, and geographic distribution to meet the needs of the anticipated
number of enrollees in the service area.

Please indicate whether your MCP meets the timing standards set by the Medicaid and CHIP agency for this
documentation.

Coordination and continuity of care for all enrollees (42 C.F.R. §§ 438.208, 457.1230(c))
1.

Which provider types are authorized by the Medicaid or CHIP agency to serve as enrollee primary care
providers? (i.e., general practitioner, family physician, internal medicine physician, OB/GYN, pediatrician, or
other licensed practitioner as authorized by the state Medicaid program)
a.

Does your MCP permit each of the provider types authorized by the Medicaid or CHIP agency to
provide primary care services to serve as primary care providers?

What steps does the MCP take to promote Medicaid and CHIP enrollees’ ongoing relationship with a usual
source of primary care?

What processes are used to coordinate services for enrollees?
a.

Are there different types of care coordination mechanisms for different types of enrollees? If so, what
are these?

Are there different types of care coordination mechanisms for acute and primary services? If so, what
are these?

If your MCP establishes separate coordination of care for medical services, LTSS, and mental health and
substance abuse services, how does it ensure exchange of necessary information between care
coordinators? How does it ensure information exchange among providers?

How are staff trained in the processes and tools required to facilitate integrated medical, behavioral, care
planning, service planning, and authorization activities?

How does the MCP ensure coordination of its services with services enrollees may receive from other MCPs
or community programs and providers?
a.

How are coordination and communication ensured when an enrollee changes MCPs or transitions
between FFS and managed care?

How are coordination and communication ensured when an enrollee is a member of more than one
MCP (e.g., duals and separate dental or behavioral health plans)?

Under what circumstances may Medicaid and CHIP enrollees have direct access to specialists?

How does your MCP manage access to any specialty care services currently not provided in-network?

Does your MCP require written treatment plans to be developed for enrollees? If yes, under what
circumstances are written treatment plans required?

158 | PROTOCOL THREE

Additional coordination and continuity of care questions: LTSS (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

Does your state’s Medicaid and CHIP agency require your MCP to meet identification, assessment, and
treatment planning requirements for dually-enrolled beneficiaries who need LTSS?

Please describe how your MCP meets any LTSS assessment mechanism requirements established by your
Medicaid and CHIP agency.

What processes are used to coordinate services for enrollees who need LTSS?
a.

Are there different types of care coordination mechanisms for LTSS as compared to acute and primary
care services? If so, what are these?

If your MCP establishes separate coordination of care for LTSS, how does it ensure exchange of necessary
information between care coordinators? How does it ensure information exchange among providers?

How are staff trained in the processes and tools required to facilitate integrated medical, behavioral and
LTSS assessment, care planning, service planning, and authorization activities for enrollees who need
LTSS?

How are coordination and communication ensured when an LTSS enrollee is a member of more than one
MCP?

Additional coordination and continuity of care questions: SHCN (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

How are “individuals with special health care needs” defined by the state Medicaid and CHIP agency?
a.
b.

Has your MCP developed any other operational definition or definitions of individuals with special health
care needs?
If yes, what is/are these and how were they developed? How do they differ from the state definition?

Does the state Medicaid or CHIP agency require your MCP to screen Medicaid and CHIP enrollees to
identify those with special health care needs?

How are individuals with special health care needs—including both individuals with special health care
needs identified by this MCP and those identified by the state Medicaid or CHIP agency or its agent—
identified and tracked within your MCP?

Does the state Medicaid agency require your MCP to assess and provide treatment plans for Medicaid and
CHIP enrollees with special health care needs? If yes, how are these activities conducted?

Disenrollment (42 C.F.R. § 438.56 and 457.1212)
1.

For what reasons may your MCP request the disenrollment of an enrollee?
a.

Has your MCP requested to disenroll an enrollee for any other reason?

Is your MCP allowed to process enrollee disenrollment requests? If so, for what reasons have enrollees
requested disenrollment?

Are enrollees required to seek redress through your MCP’s grievance system before a disenrollment request
is determined? If so, what timeframes are used by your MCP to process those grievances?

Enrollee right to information (42 C.F.R. §§ 438.100 and 457.1220; 42 C.F.R. §§ 438.10 and 457.1207)
1.

How does your MCP provide written notice of any change (that the state defines as “significant”) to the
information contained in the enrollee handbook, at least 30 days before the intended effective date of the
change?

PROTOCOL THREE | 159

How does the state define “significant”?

Have you made any such “significant” changes in the last year? If yes, what were those changes?

How do you ensure that your staff and affiliated providers comply with federal and state laws that apply to
enrollee rights?

What information is routinely provided to Medicaid and CHIP enrollees?
a.

What is the process for disseminating information to new and existing enrollees?

How often is information distributed to existing enrollees?

In what format is this information presented?

Please describe or provide copies of the formats in which information is presented to enrollees.

In what languages or alternative formats are enrollee materials and information presented? If other
languages or alternative formats are used, how was it determined that materials were needed in different
languages or formats?

Does the MCP provide written materials in alternative formats for the visually impaired? If yes, how did the
MCP determine that materials were needed for the visually impaired?

Please describe the procedures for handling calls to the MCP from non-English speaking enrollees.
a.

What instruction or guidance is available for providers that may need interpretation assistance to
provide care and services to assigned enrollees?

To what extent is the MCP responsible for responding to requests for information for potential Medicaid and
CHIP enrollees?

How does the MCP inform enrollees (and potential enrollees, if applicable) about how to obtain oral
interpreter services if they have limited proficiency in English?

10. Are there any benefits that an enrollee is entitled to under the Medicaid and CHP program, including LTSS
benefits, but that are not made available through the MCP contract? If yes, what are those benefits? How
are enrollees made aware of the Medicaid and CHIP program benefits that are outside the scope of services
available through the MCP?
11. How does the MCP ascertain the primary language spoken by the individual Medicaid and CHIP enrollees?
12. Are enrollees provided with a listing of primary care providers? If yes, does this listing include providers’
non-English language capabilities?
13. Does the MCP give written notice of the termination of a contracted provider to enrollees who receive
primary care from, or are seen on a regular basis by, the terminated providers? If yes, how is this
accomplished? Have you had to make any such notifications in the last year?
14. How does the MCP ensure that information and instructional materials intended for enrollees and potential
enrollees are easily understood by those with a variety of cognitive and intellectual capabilities?
15. How does the MCP provide its enrollees information about provider appeal rights regarding coverage of a
service?
16. Does the MCP provide information to providers on where to refer enrollees who are having difficulty
understanding the materials that have been provided to them by the MCP?

160 | PROTOCOL THREE

17. What protocols does the MCP follow to develop materials that are readily understandable by enrollees?
18. Does the MCP require providers to have access to oral interpreter services?
19. Does the MCP provide providers with guidance or assistance in accessing interpreter services if necessary?
Enrollee right to respect, dignity, privacy (42 C.F.R. §§ 438.100 457.1220)
1.

How does the MCP ensure that its own facilities and those of its affiliated providers comply with enrollee
rights such as treatment with respect, dignity, and consideration for privacy and confidentiality of
information? Please provide an example?
a.

Are there any additional considerations made for providers of LTSS, or other specialized providers,
where services may be of a more intimate nature or occur in a more isolated setting? Please provide an
example?

What processes are in place to ensure that staff members observe the MCP’s policies and procedures on
privacy and confidentiality of enrollee information?

What does the MCP do to educate staff about policies on nondiscriminatory and culturally appropriate
behavior towards enrollees?
a.

How do you monitor staff compliance with these policies?

Enrollee right to receive information on available treatment options (42 C.F.R. §§ 438.100, 438.102; and 42
C.F.R. §457.1220 and 457.1222)
1. How does the MCP ensure that providers share information on available treatment options and alternatives
with enrollees?

Does this include alternatives and options that are both within and outside the Medicaid or CHIP
contract scope of benefits?

How does the MCP ensure providers share information about HCBS as alternatives to institutional
care?

What steps does the MCP take to ensure that enrollees receive information on available treatment options
and alternatives in a manner appropriate to their condition and ability to understand?

Enrollee right to participate in decisions regarding his/her health care and advance directives (42 C.F.R. §§
438.100 and 42 C.F.R. § 438.6; 42 C.F.R. § 457.1220)
1.

How does the MCP facilitate enrollee participation in care and treatment decisions? Please describe. Could
you provide an example?

Does the MCP have any limitations in implementing federal and state laws that apply to advance directives?
If so, what are these limitations?

Enrollee rights (42 C.F.R. §§ 438.10 and 457.1207) and Enrollee information (42 C.F.R. § 438.100 and 457.1220,
and 42 C.F.R. § 438.206-210 and 457.1230(a–d))
1.

Does the MCP provide information to providers on where to refer enrollees who are having difficulty
understanding the materials that have been provided to them by the MCP?

What protocols does the MCP follow to develop materials that are readily understandable by enrollees?

Does the MCP require providers to have access to oral interpreter services?

Does the MCP provide providers with guidance or assistance is accessing interpreter services if necessary?

PROTOCOL THREE | 161

Compliance with other federal and state laws (42 C.F.R. § 438.100 and 457.1220, and 42 C.F.R. § 438.206-210
and 457.1230(a–d))
1.

What steps do MCP leaders take to ensure compliance with federal and state laws on enrollee rights?

Has the MCP ever been found non-compliant with any federal and state laws on enrollee rights? If yes, in
what area? What steps were taken to clear the violation?

If a provider/contractor/sub-contractor is found to be in violation of any federal and state laws on enrollee
rights, how does the MCP respond?

To what extent does the MCP orient new staff to federal and state laws on enrollee rights that must be
observed during day-to-day operations?
a.

How does the MCP remind staff of the importance of observing these laws during interactions with other
employees and with enrollees?

Please describe the steps taken by the MCP when staff report, or are involved in a violation of federal or
state laws on enrollee rights.

Coverage and authorization of services, including emergency and post-stabilization services (42 C.F.R. §
438.210 and 457.1230(d) and 42 C.F.R. §§ 438.114 and 457.1228)
1.

What percent of emergency department care utilized by your Medicaid and CHIP enrollees is for non-urgent
care?

Has your MCP investigated a potential relationship between inappropriate emergency department use and
enrollee access to routine and urgent care, or reviewed the most frequent diagnoses resulting in
inappropriate emergency department use?

What was the rate of denied claims for emergency and post-stabilization services in the most recent year?

What was the rate of appeals for denied claims for emergency and post-stabilization services in the most
recent year?
a.

Of these appeals, what was the rate in which claim denials were overturned?

What is the average wait time for MCP enrollees who see emergency services?

How many urgent care clinics with non-traditional hours are in the MCP’s network?

How does the MCP inform enrollees of emergency coverage?

Are emergency back-up plans created for all enrollees? If not, how is the need for an emergency back-up
plan determined? How is the emergency back-up plan shared with all appropriate parties?

Are certain LTSS providers or other specialized providers/provider types contracted specifically for afterhours/urgent/emergent need? If so, what types? How were these types determined?

10. How does the MCP ensure that it provides services in a sufficient amount, duration, and scope consistent
with contract requirements?
11. What are the MCP’s policies on service and drug limitations? What services or drugs does it limit?
a.

How does the MCP ensure that limited services can still reasonably achieve their purpose?

162 | PROTOCOL THREE

How does the MCP ensure that services supporting individuals with ongoing or chronic conditions, or
who require LTSS, are authorized in a manner that reflects the enrollee’s ongoing need for such
services and supports?

How does the MCP ensure that family planning services are provided in a manner that protects the
enrollee’s freedom to choose their preferred method?

12. How does the MCP ensure that it providers all medically necessary services specified by the contract?
13. What mechanisms does the MCP use to ensure consistent application of authorization decision review
criteria?
14. What mechanisms does the MCP use to notify providers and enrollees of adverse benefit determinations?
a.

What timeframes does the MCP use to process standard and expedited authorization decisions?

15. What notice methods does the MCP use for outpatient drug authorization decisions?
Provider Selection and Non-Discrimination (42 C.F.R. §§ 438.214 and 457.1233(a) and 42 C.F.R. §§ 438.12 and
457.1208)
1.

What is the basis or criteria used to determine individual provider participation in the MCP’s network?

What is the basis or criteria used to determine institutional or other non-individual practitioner (including
LTSS) participation in the MCP’s network?

What types of providers are subject to the MCP’s credentialing process?
a.

How are provider qualifications (including background check requirements) verified for provider types
not subject to the credentialing process?

Please describe the provider credentialing process used by the MCP.

What steps does the MCP take to ensure that it does not employ or contract with providers who have been
excluded from participation in federal health care programs?

What steps does the MCP take to ensure that providers who serve high-risk or costly populations are not
discriminated against in the selection process, and when considering reimbursement and indemnification?

What criteria is the basis for denial of provider participation in the MCP’s network?

Sub-Contractual Relationships and Delegation (42 C.F.R. § 438.230 and 42 C.F.R. § 457.1233(b))
1.

What services and activities are delegated to and performed by sub-contractors?

Please describe the MCP’s process for identifying and selecting contractors. How is it determined that a
contractor has the ability to provide the sub-contracted services?

Please describe how your MCP assesses the quality of sub-contracted services and sub-contractor
compliance with federal, state and contractual requirements.

Practice guidelines: adoption (42 C.F.R. §§ 438.236(b) and 457.1233(c))
1.

What organizational component of your MCP is responsible for the adoption of practice guidelines used by
your MCP?

How does your MCP establish priorities for adoption of practice guidelines?

PROTOCOL THREE | 163

How does your MCP consider the enrolled Medicaid and CHIP population’s health needs in the
adoption of practice guidelines?

What guidelines has your MCP adopted?

By what institutional process were they adopted?

To what extent are your MCP’s guidelines “evidence-based”? By evidence-based, we mean systematically
developed statements to assist practitioner and patient decisions about appropriate health care for specific
clinical circumstances.

How does your MCP consider the enrolled Medicaid and CHIP population’s health needs in the adoption of
practice guidelines?

How are affiliated providers consulted as guidelines are adopted and re-evaluated?

What mechanism(s) does your MCP have for periodically evaluating and updating the guidelines it has
adopted?

Practice guidelines: dissemination and application (42 C.F.R. §§ 438.236(c) and 457.1233(c))
1.

How are practice guidelines disseminated to providers?

When and how are guidelines disseminated to enrollees and potential enrollees?

To what extent are the practice guidelines adopted by your MCP a component of your MCP’s Quality
Assessment and Performance Improvement (QAPI) program?

Is there a process in place to ensure communication between those responsible for the QAPI program and
the practice guidelines adoption process?

What steps are taken to ensure that decision-making in the areas of utilization management or coverage
determinations and other functional areas are consistent with the adopted practice guidelines?

Health information systems (42 C.F.R. §§ 438.242 and 457.1233(d))
1.

Describe the types of data collection systems that are in place to support the clinical and administrative
operations of your MCP. Specifically, what data is routinely collected to support utilization management,
grievance systems, and enrollment services?

What processes are in place to obtain data from all components of your network (e.g., health care facilities,
physician, laboratories, and LTSS, and other specialized providers)?

To what extent does your MCP require and receive data in standardized formats?

Are there any components of your network from which you do not receive standardized (or any)
information on services?

How are enrollee and provider data collected and integrated across all components of your MCP’s network?
a.

How is this used to produce comprehensive information on enrollee needs and utilization and to
otherwise support management?

II. Quality Assessment and Performance Improvement Program
Quality assessment and performance improvement program: general rules and basic elements (42 C.F.R. §§
438.330 and 457.1240(b))

164 | PROTOCOL THREE

Does the state require the MCP to address a specific topic or topics in your performance improvement
projects? If yes, what types of projects are required? For each PIP, at a minimum, include how significant
improvement was measured, how improvement will be/was sustained, and how beneficiary health outcomes
and satisfaction will be/was measured, and how the intervention will/has improved access and/or quality of
care?
a.

For duals-only MCPs, was a Medicare Advantage PIP substituted for a state-required PIP?

Has CMS specified any specific PIPs? If yes, what types of projects are required? For each PIP, at a
minimum, include how significant improvement was measured, how improvement will be/was sustained,
and how beneficiary health outcomes and satisfaction will be/was measured, and how the intervention
will/has improved access and/or quality of care?

Does the state require your MCP to collect and submit performance measures or to submit data to the state
for it to calculate performance measures? If yes, what performance measures are specified by the state and
who calculates each measure, the MCP or the state?
a.

If CMS specifies any performance measures, what performance measures are collected and submitted,
if any?

If the MCP provides LTSS, what LTSS performance measures are collected and submitted, including
but not limited to measures of quality of life, rebalancing institutional and community-based services and
community integration activities?

How does the MCP detect over- and under-utilization? Please provide examples of how your quality
assessment and improvement program has monitored to detect under- and over-utilization. What standards
and measures are used?

How does the MCP define enrollees with “special health care needs”? Does this definition match the state’s
definition of special health care needs? How are these enrollees identified/ tracked within your MCP?

How does the MCP assess the quality and appropriateness of care including LTSS, furnished to enrollees
with special health care needs? Please provide examples.

Does the state require the MCP to evaluate the impact and effectiveness of its quality assessment and
performance improvement program?

How does the MCP conduct its evaluation? What aspects of the program are included in the
evaluation?

How often does the MCP conduct its evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of the findings?

What is reported to the state, and how often?

For MCPs that provide LTSS services:
a.

How does the MCP assess quality and appropriateness of care in general, including but not limited to
between care settings and comparing treatment plans to service/supports received?

How does the MCP participate in the state’s efforts to prevent, detect, and remediate critical incidents?

What interventions are used or are anticipated to be used to improve LTSS quality? How will the
interventions be evaluated for effectiveness? How will improvement be sustained or increased?

PROTOCOL THREE | 165

Quality assessment and performance improvement program: program review by the state (42 C.F.R. §§
438.330(e) and 457.1240(b))
1.

How does the state review the impact and effectiveness of the MCP’s QAPI program, including outcomes
and trended results from the PIPs, reporting on performance measures, and the results of community
integration for beneficiaries receiving LTSS?
a.

What is the MCP’s role in the state’s evaluation?

What information, if any, does the MCP provide to the state?

What feedback, if any, does the MCP receive from the state? How does the MCP implement the
feedback?

III. Grievance System
Grievance system: denial of services (42 C.F.R. § 438.228
1.

How does the MCP track requests for covered services that the MCP or its providers have denied?

What was the volume of denied claims for services in the most recent year?

How do you ensure that Medicaid enrollees who were denied services were notified of their right to a state
fair hearing?

Grievance system: general requirements (42 C.F.R. §§ 438.402 and 457.1260)
1.

Who in the MCP is responsible for the development and oversight of the appeals and grievance resolution
process and access to state fair hearings or review?

What have been the volume of appeals/grievances/requests for state fair hearings or reviews in the past
year and the most common areas of concern expressed by Medicaid and CHIP enrollees?
a.

How has the MCP addressed these concerns?

Describe the notice and appeals process for adverse actions on enrollee requests for services or payment.
Please describe the particular steps, including time frames.

Grievance systems: continuation of benefits (42 C.F.R. § 438.404(b)(6))
1.

Does the Medicaid enrollee's right to have benefits continue pending resolution of the appeal, the process to
request that benefits be continued, and the circumstances under which the enrollee may be required to pay
the costs of these services differ between medical and LTSS? Note that continuation of benefits
requirements do not apply to CHIP enrollees.
a.

If so, how?

Are there any special considerations required for continuation of LTSS pending resolution of an appeal?

Handling of grievances and appeals (42 C.F.R. §§ 438.406 and 457.1260)
1.

To what extent does your MCP provide Medicaid and CHIP enrollees with assistance in completing forms
and taking other procedural steps in the grievance and appeal process? How does the MCP provide
assistance?

How does your MCP treat oral requests by Medicaid and CHIP enrollees to appeal actions?

166 | PROTOCOL THREE

As part of an appeal, to what extent do enrollees and their representatives have an opportunity to:
a.

Present evidence, and

Examine the enrollee’s case file, including medical records, and any other documents and records
considered during the appeals process.

What are the qualifications and credentials of individuals who make decisions on grievances and appeals?
a.

How does the MCP ensure that these individuals have not been involved in any previous level of review
or decision-making?

How does the MCP ensure that these individuals have the appropriate clinical expertise in treating the
enrollee’s condition or disease, if deciding any of the following:
i.

An appeal of a denial that is based on lack of medical necessity

ii.

A grievance regarding denial of expedited resolution of an appeal

iii.

A grievance or appeal that involves clinical issues

Is there a process in place to monitor either the appeal and grievance process or the areas of concern
identified by enrollee appeals and grievances?

Resolution and notification: grievances and appeals (42 C.F.R. §§ 438.408 and 457.1260)
1.

Approximately how many grievances did the MCP receive in the most recent reporting year?

Approximately how many appeals did the MCP receive in the most recent reporting year?

Approximately what percent of notices of action on requests for service authorization or payment by
Medicaid and CHIP enrollees are appealed to the MCP?

Approximately what percent of notices of action on requests for service authorization or payment by
Medicaid and CHIP enrollees are appealed to the state fair hearing process?
a.

Approximately what percent of these are overturned by the state?

Expedited resolution of appeals (42 C.F.R. §§ 438.410 and 457.1260)
1.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of an appeal? If so, please describe this process. What are the time frames for this process?

Are physicians allowed to request expedited appeals on behalf of an enrollee? How does the MCP protect
physicians who make such requests?

Information about the grievance system to providers and subcontractors (42 C.F.R. §§ 438.414 and 457.1260)
1.

Who in your MCP has responsibility for the functioning of the grievance process and the authority to require
corrective action?

Did your state Medicaid and CHIP agency develop or approve the description of your MCP’s grievance
system provided to Medicaid and CHIP providers? [Note: clarify if the state Medicaid and CHIP agency
developed or approved] If it approved your description, how is the state’s approval documented?

PROTOCOL THREE | 167

Recordkeeping and reporting requirements: grievances and appeals (42 C.F.R. §§ 438.416 and 457.1260)
1.

Where in your MCP are records on Medicaid and CHIP enrollee grievances and appeals kept?

168 | PROTOCOL THREE

MCP Information Systems Staff Interview
Instructions. This interview will assess the MCP’s information management function and how it supports the other
functions of the organization, such as planning and operations, quality assessment and improvement program
activities, care coordination, etc. This is also an opportunity to explore the extent to which the health information
needs of the entire MCP and provider network are measured, assessed, and improved.
The interview should include MCP staff responsible for health information systems issues at the MCP. It should
include those responsible for technology implementation, as well as staff that are responsible for the information
quality, information transmittal, information sharing, and information policy and procedure development and
implementation.
Information system capabilities. The interviewees should receive a copy of the MCP’s most recent Information
Systems Capability Assessment (ISCA) (see Appendix A) that has either been completed by an independent
organization reviewing the MCP or has been completed by the organization conducting this compliance review. 61 The
findings of the ISCA will serve as a guide to conducting this interview. During this interview, validate the information
provided about the MCP on the ISCA, explore any areas of concern, and gather missing or additional information for
use in evaluating standards compliance, paying particular attention to how data are defined and captured across the
MCP and how data transmission and integration takes place across the MCP. Questions and areas of discussion
should be based on the findings of the ISCA, and may include:
1.

Are the findings of the most recent assessment of the MCP’s information systems capacity reflective of your
own assessment of capabilities?

What are your information system’s strengths and weaknesses?
a.

What has the MCP done to address information system problem areas?

What information needs does your MCP have that are not currently met by your present information system?
a.

What has the MCP done to address these needs?

Is the data collected from network providers on services to enrollees subject to accuracy and timeliness
checks?

Please describe procedures used to screen all data, both internal and external, for completeness, logic, and
consistency.

How is enrollee-specific data and information made available when and where needed by the MCP’s
provider network?

Delivery network (42 C.F.R. §§ 438.206 and 457.1230(a))
1.
2.

How does your information system track services provided by and/or reimbursed to out-of-network
providers?
Describe the capabilities to routinely collect data on use of out-of-network providers (excluding Point of
Service-related use).
a.

Is data on use of out-of-network providers separately available for Medicaid and CHIP enrollees?

Assurances of adequate capacity and services (42 CFR §§ 438.207(b) – (c) and 457.1230(b))

PROTOCOL THREE | 169

Please describe any information system capabilities used to demonstrate to the Medicaid and CHIP agency
that the MCP offers an appropriate range of preventive, primary care, specialty services, and LTSS that is
adequate for the anticipated number of enrollees in your service area.

Please describe any information system capabilities used to demonstrate to the Medicaid and CHIP agency
that the MCP maintains a network of providers that is sufficient in number, mix, and geographic distribution
to meet the needs of the anticipated number of enrollees in the service area.

Coordination and continuity of care for all enrollees (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

How does the MCPs information system integrate medical, behavioral and LTSS assessments, care
planning, service planning and authorization information and processes?

Health information systems (42 C.F.R. §§ 438.242 and 457.1233(d))
1.

How is the data collected from network providers on services to enrollees checked for accuracy and
timeliness?

Please describe procedures used to screen all data, both internal and external, for completeness, logic, and
consistency.

170 | PROTOCOL THREE

Quality Assessment and Performance Improvement Program Staff Interview
Instructions. This interview with quality improvement program leaders and staff provides an opportunity to gain a
more thorough understanding of the approaches and processes used by the MCP to assess and improve quality.
Availability of services (42 C.F.R. §§ 438.206 and 457.1230(a))
1.

What information is generated through QAPI activities to assess the MCP’s availability of services?
a.

What issues were considered in the assessment process?

What services, such as family planning and women’s health services, have QAPI activities focused on?

Please describe the assessment results.
a.

Are there any service-specific results? If so, please describe them.

Has the MCP implemented QAPI findings relevant to the availability of services? If so, please describe them
and their results.

How frequently does the MCP evaluate the volume and enrollee access to LTSS services? What factors are
used in evaluation of the LTSS network? Note that this is not applicable to CHIP.

Furnishing of services-timely access (42 C.F.R. §§ 438.206(c) and 457.1230(a))
1.

Please describe any recent QAPI activities implemented to monitor the MCP’s compliance with its
established standards for timeliness of access to care and member services.
a.

Please describe any recent QAPI activities implemented to promote cultural competency and delivery of
services in a culturally competent manner.
a.

What were the results of these QAPI activities?

What are the results of these QAPI activities?

Please describe any recent QAPI activities implemented to promote physical access, reasonable
accommodations, and accessible equipment for Medicaid enrollees with physical or mental disabilities.
a.

What are the results of these QAPI activities?

Enrollee rights (42 C.F.R. §§ 438.100 and 457.1220)
1.

How is the enrollee’s right to be free from restraint or seclusion monitored for enrollees, including, for
example, those receiving LTSS? Note that requirements applying to LTSS are not applicable to CHIP.

Provider selection (42 C.F.R. §§ 438.214 and 457.1233(a))
1.

What type of information is generated through the quality improvement program to support re-credentialing
of individual practitioner providers?

What types of information does the quality improvement program provide to support the re-credentialing of
institutional and other non-practitioner providers?

What types of information does the quality improvement program provide to support the evaluation of LTSS
provider qualifications?

PROTOCOL THREE | 171

Practice guidelines (42 C.F.R. §§ 438.236 and 457.1233(c))
1.

Through what process does your MCP ensure necessary communication occurs between those responsible
for the QAPI program and the administrative function responsible for adopting practice guidelines?

Quality assessment and performance improvement program (42 C.F.R. §§ 438.330 and 457.1240(b))
1.

Does the state require the MCP to address a specific topic or topics and/or indicators in your performance
improvement projects? If yes, what types of projects are required?

How does the MCP define enrollees with “special health care needs”? How are these enrollees identified/
tracked within your MCP?

How does the MCP assess the quality and appropriateness of care including LTSS, furnished to enrollees
with special health care needs? Please provide examples.

Does the MCP evaluate the effectiveness of its quality assessment and performance improvement program?
How often?

Please describe the evaluation process. What aspects of the program are included in the evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of these findings?

Does the state require the MCP to evaluate the impact and effectiveness of its quality assessment and
performance improvement program?
a.

How does the MCP conduct its evaluation? What aspects of the program are included in the
evaluation?

How often does the MCP conduct its evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of these findings?

What is reported to the state, and how often?

What is the MCP’s role in the state’s evaluation?

What information, if any, does the MCP provide to the state?

What feedback, if any, does the MCP receive from the state? How does your MCP implement the
feedback?

What evaluation findings are reported to the state and how often?

172 | PROTOCOL THREE

What interventions are used or are anticipated to be used to improve LTSS quality? How will the
interventions be evaluated for effectiveness? How will improvement be sustained or increased?

Health information systems (42 C.F.R. §§ 438.242 and 457.1233(d))
1.

How are enrollee and provider data from all components of your MCP’s network used in your MCP’s quality
assessment and performance improvement program?
a.

Are there any components in your network for which you do not have adequate enrollee utilization and
provider data?

How is data obtained from the meaningful use of certified electronic health records (EHRs) utilized as part of
the MCP’s quality improvement program?

Handling of grievances and appeals (42 C.F.R. §§ 438.406 and 457.1260)
1.

What is the process used to monitor the appeal and grievance process?

What is the process to monitor areas of concern identified by enrollee appeals and grievances?

Recordkeeping and reporting requirements on grievances and appeals (42 C.F.R. §§ 438.416 and 457.1260)
1.

To what extent is information on Medicaid and CHIP enrollee grievances and appeals analyzed and included
as part of your MCP’s Quality Assessment and Performance Improvement Program?

PROTOCOL THREE | 173

Provider/Contractor Services Staff Interview
Instructions. This is an interview of MCP staff members who are responsible for establishing and maintaining
communications with the MCP’s individual practitioners and other types of health care providers (e.g., organizations).
This includes staff responsible for management of the credentialing process and oversight of delegated activities.
Through these interviews, the reviewer(s) will assess enrollee rights; the credentialing and appointment process;
oversight of the providers; and how information is communicated to providers.
Enrollee rights (42 C.F.R. §§ 438.100 and 457.1220 and 42 C.F.R. §§ 438.206-210 and 457.1230(a-d)))
1.

How does the MCP inform its providers (individual, institutional, and LTSS providers) about enrollee rights
and responsibilities?
a.

How does the MCP monitor for compliance with these rights by its providers?

To what extent, if any, does the MCP supply providers with information on where to refer enrollees who are
having difficulty understanding the materials that have been provided to them by the MCP?

Does the MCP require providers to have access to oral interpreter services?
a.

Does the MCP supply providers with guidance or assistance in accessing oral interpreter services if
necessary?

How does the MCP ensure that its own facilities and those of its affiliated providers comply with enrollee
rights to treatment with respect, dignity, and consideration for privacy?
a.

Are there any additional considerations made for providers of LTSS, where services may be of a more
intimate nature or occur in a more isolated setting? Please provide examples.

How does the MCP ensure that enrollees are not discriminated against in its own facilities and those of its
affiliated providers when seeking health care services consistent with their covered benefits?

Please describe the MCP’s credentialing, verification and oversight process for primary care providers, other
health care professionals, LTSS and institutional providers.
a.

What is encompassed by reviews and evaluations of these providers?

Do these processes involve visits to the providers’ care delivery sites?

What methods are used to encourage providers to share information on available treatment options and
alternatives with enrollees?

What processes are in place for monitoring providers to determine that they are providing information on
available treatment options and alternatives?

What requirements does the MCP have for providers/contractors relative to enrollee advance directives?
a.

How is it determined that providers/contractors are meeting the MCP’s requirements?

10. How does the MCP inform all of its network providers, including its LTSS, individual and institutional
providers, about enrollee rights to service availability, coordination and continuity of care, coverage and
authorization of service, and to obtain a second opinion from an appropriately qualified health professional?
a.

How does the MCP monitor for compliance with these rights by its providers?

11. How are the MCP’s network providers informed of enrollees’ right to request and receive a copy of their
medical records, and to request that they be amended or corrected?

174 | PROTOCOL THREE

12. What steps does the MCP take to ensure that providers/contractors are aware of and in compliance with
applicable federal and state laws on enrollee rights?
13. If a provider/contractor is found in violation of a federal or state law concerning enrollee rights, what action is
taken by the MCP?
Availability of services (42 C.F.R. §§ 438.206 and 457.1230(a))
1.

Please describe the MCP credentialing and re-credentialing process.
a.

Is the process different for Medicaid and CHIP providers than for providers serving other networks? If
yes, what are the differences?

How is it determined that providers are geographically accessible to Medicaid and CHIP enrollees and
physically accessible to enrollees with disabilities?

Please describe the processes for monitoring the provider network to determine that Medicaid and CHIP
requirements about timeliness, availability, and accessibility are being met.
a.

What are the most recent findings from this process?

How often in the last year has your MCP had to arrange for services or reimbursements to out-of-network
providers?

How does the MCP evaluate the expected utilization of institutional care in comparison with the use of
HCBS as an alternative?

Does the MCP maintain accessibility information on its LTSS and other specialized providers? If yes, how is
this maintained and shared with enrollees?

How does the MCP encourage the promotion of culturally competent service delivery by LTSS and other
specialty providers?

Are there any limits to choice of LTSS and other specialty providers?

Timely access to service (42 C.F.R. §§ 438.206(c)(1) and 457.1230(a))
1.

Ask only if MCP is a MCO, PIHP, or PAHP: Are your MCP’s provider services available 24 hours a day, 7
days a week, when medically or otherwise necessary to meet the enrollee’s needs?
a.

Are certain LTSS or other specialized providers/provider types contracted specifically for afterhours/urgent/emergent need?

If yes, what types? How were these types determined?

Are providers included in developing beneficiary emergency back-up plans? If they are not involved in the
back-up plan development, how are they made aware of their responsibility for emergency back-up?

Are the hours of operation of the provider network serving Medicaid and CHIP enrollees different from the
hours of operation of the provider network serving other enrollees? If yes, why are they different?

Does the MCP continuously monitor its provider network for compliance with established standards on
timeliness of access to all care and member services? If yes, how, and what are the most recent findings?

What steps are taken to address provider non-compliance with established standards for timeliness of
access to care and member services?

PROTOCOL THREE | 175

How are corrective actions assessed for effectiveness? Please describe the follow up and monitoring.

Coordination and continuity of care for all enrollees (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

How are primary care providers serving enrollees with special health care needs made aware of and involved
in procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Assuring appropriate use of specialists?

Coordinating primary care services with care provided by other MCOs, PIHPs or PAHPs serving the
enrollee?

Coordinating care with other providers, including specialist and LTSS providers?

Additional coordination and continuity of care questions: SHCN (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

How are specialty providers serving enrollees with special health care needs made aware of and involved in
procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Coordinating specialty care services with care provided by other MCPs serving the enrollee?

Coordinating care with other providers, including primary and LTSS providers?

How are LTSS providers serving enrollees with special health care needs made aware of and involved in
procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Coordinating care with other providers, including primary and LTSS providers?

Additional coordination and continuity of care questions: LTSS (42 C.F.R. § 438.208
1.

How are LTSS providers serving enrollees with special health care needs made aware of and involved in
procedures for:
a.

Assessing individuals with special health care needs?

Ensuring that treatment plans address the needs identified by the assessment?

Coordinating care with primary care and specialty providers?

Coverage and authorization of services (42 C.F.R. §§ 438.210 and 457.1230(d))
1.

Do contracts/agreements with individuals or organizations performing utilization review offer any
performance incentives? If yes, please describe the incentives. [Note to reviewers: Look for any incentives
for denying, limiting, or discontinuing authorization of services.]

176 | PROTOCOL THREE

Are network providers notified of the information ordinarily required to process an authorization request?

Please describe the process for notifying the requesting provider of any decision to deny, limit, or
discontinue authorization of services.
a. What are the MCP’s time frames for notification?

Does the MCP contract with all LTSS and other specialized provider types identified in the state’s benefit
package? If not, what provider types are not contracted? How are enrollees’ needs met in lieu of this
service availability?

Are there any universal service limitations on LTSS? If yes, what are the service limitations, and how were
these determined?

Provider selection (42 C.F.R. §§ 438.214 and 457.1233(a))
1.

What types of individual practitioners are subject to the MCP’s credentialing process?

Please describe the MCP’s credentialing processes for individual practitioners.
a. How often does this process take place?
b. What items of credentials information are updated during the process?
c. Are site visits made to providers? When and how often? How is it determined that a site visit will be
made?
d. Who is involved in the MCP’s credentialing activities?

Please describe the MCP’s re-credentialing processes for individual practitioners.
a. What types of information are monitored and reviewed during the re-credentialing process?
b. What other operations of the MCP contribute information to be used in the re-credentialing process?

Ask only if MCP is a MCO, PIHP, or PAHP: Please describe the MCO’s /PIHP’s/PAHP’s processes for
selecting and monitoring institutional and other non- practitioner network providers (including LTSS).
a. What information is reviewed as a part of this process?
b. Are site visits made? When and how often?

Please describe the MCP’s credentialing and re-credentialing processes for institutional providers.
a. Are site visits a part of the process to credential and re-credential institutional providers?
b. How frequently is re-credentialing performed?
c. What items of information are typically reviewed during the evaluation and reevaluation process?

What other MCP operations contribute to the evaluation of a network institutional provider?

What criteria is the basis for denial of provider participation in the MCP’s network?

How does the MCP verify the skills and requirements of LTSS providers, including self-directed support
options? (i.e., background checks, exclusions, certifications and/or licensures)

PROTOCOL THREE | 177

Grievance systems (42 C.F.R. § 438.228)
1.

Please describe the process for notifying the requesting provider of any decision to deny, limit, or
discontinue authorization of services.
a. What are the MCP’s time frames for notification?

Sub contractual relationships and delegation (42 C.F.R. §§ 438.230 and 457.1233(b))
1.

What types of activities are performed by (and thereby delegated to) contractors?

Please describe your MCP’s process for identifying and selecting contractors.
a.

What steps does your MCP take to determine that an entity to which functions will be delegated is capable
of performing the delegated functions?
a.

How is it determined that a contractor has the ability to perform the activities that are being delegated by
the MCP?

Please describe any evaluation process that your MCP has in place.

For each of the activities that have been delegated:
a. Is there any ongoing monitoring and review of entities performing delegated activities?
i.

How this is accomplished?

ii.

Is the process the same for all delegates at all times?

iii.

Are there any instances when your MCP varies the monitoring process or the timing of
evaluation?

b. Does your MCP perform an annual evaluation of the delegate’s sub-contractor’s performance?
i.

Please describe the process to conduct this evaluation. What is included in the evaluation?

c. What is done with the results of delegate evaluations?

Do the results of the most recent delegate subcontractor evaluations specify any necessary
corrective action for problems or deficiencies identified?

ii.

Please describe some of the recommendations made to delegates in an effort to improve
performance.

What steps does your MCP take to assure that the delegate implements corrective actions?

Who in the MCP is assigned responsibility for monitoring the delegate’s performance?

Does the MCP delegate any of its activities to MLTSS providers? If yes, how is the provider’s ability to carry
out delegated activities determined and monitored?

Practice guidelines (42 C.F.R. §§ 438.236 and 457.1233(c))
1.

What mechanism is in place to consult affiliated providers as practice guidelines are adopted and reevaluated?

How are practice guidelines disseminated to providers?

178 | PROTOCOL THREE

Quality assessment and performance improvement program (42 C.F.R. § 438.330)
1.

How does the MCP monitor LTSS provider quality, appropriateness of care, compliance with state and plan
requirements, and enforce corrective action when necessary?

Please describe any QAPI activities implemented to assess or improve communications with the MCP’s
providers.
a.

What are the results of these activities?

Please describe any QAPI activities implemented to assess or improve the credentialing process and
oversight of the MCP’s delegated activities.
a.

What are the results of these activities?

Health information systems (42 C.F.R. §§ 438.242 and 457.1233(d))
1.

Does the MCP have data collection requirements for LTSS providers, health care facilities, and physicians?
a.

How are the requirements communicated to these organizations and individuals?

If issues arise in the timeliness and accuracy of the data that is being collected and submitted, who notifies
the health care facility or physician?

Information about the grievance system to providers and subcontractors (42 C.F.R. §§ 438.414 and 457.1260)
1.

When are providers given information about the MCP’s Medicaid and CHIP complaint and grievance
system?
a.

What is typically included in the information given to providers relative to Medicaid and CHIP
grievances?

PROTOCOL THREE | 179

Enrollee Services Staff Interview
Instructions. The enrollee services staff interview provides an opportunity to speak with MCP staff members who are
responsible for communicating with enrollees. Relevant staff includes those individuals responsible for written
communication, phone responses to inquiries and problems, the complaint and grievance system and other services
designed to assist Medicaid and CHIP enrollees in their use of MCP services. Through this interview, the EQRO will
assess the manner in which the MCP and its provider network address issues relating to the rights of enrollees; the
MCP’s efforts regarding enrollee education and communication; the mechanisms in place to insure that information
needed to provide services to enrollees is available throughout the MCP; and the aspects of enrollee services are
measured, how collected data is assessed, and what efforts have been made to improve enrollee services.
Enrollee right to information (42 C.F.R. §§ 438.100 and 457.1220; 42 C.F.R. §§ 438.10 and 457.1207)
1.

What information is routinely provided to Medicaid and CHIP enrollees?
a.

What is the process for disseminating information to new and existing enrollees?

How often is information distributed to existing enrollees?

In what format is this information presented?

Please describe or provide copies of the formats in which information is presented to enrollees.

In what languages or alternative formats are enrollee materials and information presented? If yes, how was
it determined that materials were needed in different languages?

Does the MCP provide written materials in alternative formats for the visually impaired? If yes, how did the
MCP determine that materials were needed for the visually impaired?

Please describe the procedure for handling calls to the MCP from non-English speaking enrollees.
a.

What instruction or guidance is available for providers that may need interpretation assistance to
provide care and services to assigned enrollees?

To what extent is the MCP responsible for responding to requests for information for potential Medicaid and
CHIP enrollees?

How does the MCP inform enrollees (and potential enrollees, if applicable) about how to obtain oral
interpreter services if they have limited proficiency in English?

Are there any benefits that an enrollee is entitled to under the Medicaid and CHIP program, including LTSS
benefits, but that are not made available through the MCP contract? If yes, what are those benefits? How
are enrollees made aware of the Medicaid and CHIP program benefits that are outside the scope of services
available through the MCP?

How does the MCP ascertain the primary language spoken by the individual Medicaid and CHIP enrollees?

10. Are enrollees provided with a listing of primary care providers? If yes, does the listing include providers’
non-English language capabilities?
11. Does your MCP give written notice of termination of a contracted provider to enrollees who receive primary
care from, or are seen on a regular basis by, the terminated providers? If yes, how is this accomplished?
Have you had to make any such notifications in the last year?
12. Does your MCP give enrollees any notice of significant changes in the information in the Enrollee
Handbook? When and how does this occur? Have you had to make any such notifications in the last year?

180 | PROTOCOL THREE

13. How does the MCP ensure that information and instructional materials intended for enrollees and potential
enrollees are easily understood by those with a variety of cognitive and intellectual capabilities?
14. How does the MCP provide its enrollees information about provider appeal rights regarding coverage of a
service?
Enrollee right to respect, dignity, and privacy (42 C.F.R. §§ 438.100 and 457.1220; and 42 C.F.R. §§ 438.206210 and 457.1230(a-d))
1.

How does the MCP ensure that its own facilities and those of its affiliated providers comply with enrollee
rights to treatment with respect, dignity, and consideration for privacy and confidentiality of information?
a.

Are there any additional considerations made for providers of LTSS or other specialized services, where
services may be of a more intimate nature or occur in a more isolated setting? Please provide
examples.

Enrollee right to participate in decisions regarding his or her health care (42 C.F.R. § 438.100 and 457.1220);
and regarding advance directives (42 C.F.R. §§ 438.10(g) and 457.1207; and 42 C.F.R. §§ 438.206-210 and
457.1230(d))
1.

To what extent does the MCP allow enrollees to participate in care and treatment decisions? Please
describe some of the ways in which this is accomplished.

To what extent are Medicaid and CHIP enrollees informed at the time of enrollment of their right to accept or
refuse treatment and to execute an advance directive, and the MCP’s policies on implementation of that
right?

Enrollee right to service availability, coordination and continuity of care, coverage and authorization of
service, and to obtain a second opinion from an appropriately qualified health professional (42 C.F.R. §§
438.100 and 457.1220; and 42 C.F.R. §§ 438.206-210 and 457.1230(a-d))
1.

How does the MCP monitor compliance of enrollee rights to service availability, coordination and continuity
of care, coverage and authorization of service, and to obtain a second opinion from an appropriately
qualified health professional?
a.

What are the most recent results of this monitoring?

Enrollee right to request and receive medical records (42 C.F.R. §§ 438.100 and 457.1220; and 42 C.F.R. §§
438.206-210 and 457.1230(a-d))
1.

How do enrollees obtain access to their medical records maintained by the MCP, including records
maintained by providers/contractors from whom the enrollee has received services?

How are enrollees informed of their right to request and receive a copy of their medical records, and to
request that they be amended or corrected?

Has the MCP received any complaints about an enrollee’s inability to access their medical records in a
timely manner? If yes, what was the volume and nature of the complaints? How were they resolved?

Compliance with other federal and state laws (42 C.F.R. §§ 438.100 and 457.1220; and 42 C.F.R. §§ 438.206210 and 457.1230(a-d))
1.

Does the MCP orient staff to the federal and state laws on enrollee rights that must be observed during dayto-day operations? Does the MCP remind staff of the importance of observing these laws during
interactions with other employees and with enrollees?

Describe the procedure for handling an enrollee complaint involving a perceived violation of their rights.

PROTOCOL THREE | 181

Availability of services (42 C.F.R. §§ 438.206 and 457.1230(a))
1.

What processes does the MCP take to monitor availability and accessibility of services to Medicaid and
CHIP enrollees?
a.

What are the most recent findings from this process?

Is there any information that is routinely collected and monitored to determine that care and services are
being rendered to Medicaid and CHIP enrollees in a timely manner?
a.

What are the most recent findings of this monitoring?

Availability of services-delivery network (42 C.F.R. §§ 438.206(b) and 457.1230(a))
1.

Are Medicaid and CHIP enrollee requests for out-of-network providers tracked?
a.

How often do Medicaid and CHIP enrollees request services from out-of-network providers?

What are their reasons for requesting out-of-network providers?

How often do Medicaid and CHIP enrollees receive services from out-of-network providers?

Availability of services-Furnishing of services (42 C.F.R. §§ 438.206(c) and 457.1230(a))
1.

Ask only if MCP is a MCO/ PIHP or PAHP: Are MCO/ PIHP/PAHP and provider services available 24
hours a day, 7 days a week, when medically appropriate?

How frequently does enrollee services staff receive complaints about provider hours of operation not being
available to enrollees when medically necessary?

Does the MCP conduct surveys, focus groups or other activities to receive the feedback of Medicaid and
CHIP enrollees? If yes, what are the most recent findings about Medicaid and CHIP enrollee perceptions
about availability of MCP and provider services?

Coordination and continuity of care (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

How are Medicaid and CHIP enrollees with special health care needs—including both individuals with
special health care needs identified by your MCP and individuals identified by the state Medicaid and CHIP
agency or its agent

How does this MCP identify and assess Medicaid and CHIP enrollees with special health care needs?

What proportion of Medicaid and CHIP enrollees has an ongoing source of primary care?

Coverage and authorization of services (42 C.F.R. §§ 438.210 and 457.1230(d))
1.

How frequently does enrollee services staff receive complaints about difficulty obtaining emergency or poststabilization services?

Please describe the procedure for handling member calls regarding need for emergency services.

Enrollment and disenrollment (42 C.F.R. §§ 438.56 and 457.1212)
1.

Please describe the procedures that are followed when a request for disenrollment is received from an
enrollee.

182 | PROTOCOL THREE

How is disenrollment information tracked through or by other MCP operations (e.g., grievance process,
quality improvement, administration)?
a.

How many requests by Medicaid and CHIP enrollees were received last year for disenrollment?

What were the cited causes?

Grievance systems (42 C.F.R. § 438.228)
1.

Please describe the process for notifying Medicaid and CHIP enrollees of any decision to deny, limit, or
discontinue a request for service.
a.

What are the MCP’s time frames for notification?

Practice guidelines (42 C.F.R. §§ 438.236 and 457.1233(c))
1.

How often does your MCP receive requests from enrollees and potential enrollees for practice guidelines?
How does your MCP respond to these requests?

When and how does your MCP disseminate practice guidelines to enrollees?

Grievance system - general requirements (42 C.F.R. §§ 438.402 and 457.1260)
1.

What enrollee materials contain information about the complaint and grievance processes? When are
enrollees presented with this information?

Please describe the process for handling authorization decisions that are adverse to the enrollee.

Handling of grievances and appeals (42 C.F.R. §§ 438.406 and 457.1260)
1.

What MCP department or staff members are responsible for assisting enrollees to use the organization’s
complaint or grievance system, including completing forms, or taking other steps to resolve an appeal or
grievance? What kind of assistance is made available to Medicaid and CHIP enrollees?

What are the qualifications and credentials of individuals who make decisions on grievances and appeals?
a.

How does the MCP ensure that these individuals have not been involved in any previous level of
review or decision-making?

How does the MCP ensure that these individuals have the appropriate clinical expertise in treating the
enrollee’s condition or disease, if deciding any of the following:
i.

An appeal of a denial that is based on lack of medical necessity

ii.

A grievance regarding denial of expedited resolution of an appeal

iii.

A grievance or appeal that involves clinical issues

How does your MCP treat oral requests by Medicaid and CHIP enrollees to appeal actions?

As part of an appeal, to what extent do enrollees and their representatives have an opportunity to:
a.

Present evidence, and

Examine the enrollee’s case file, including medical records, and any other documents and records
considered during the appeals process

PROTOCOL THREE | 183

Resolution and notification: Grievances and appeals (42 C.F.R. §§ 438.408 and 457.1260)
1.

Please describe the MCP’s grievance resolution process.

Please describe the MCP’s appeal resolution process.

How is it determined that an enrollee’s appeal requires expedited resolution?

What percent of appeal resolutions that are completely or partially adverse to Medicaid and CHIP enrollees
are appealed to the state fair hearing process or review? Of these, what percent are overturned by the state
Medicaid and CHIP agency?

Expedited resolution of appeals (42 C.F.R. §§ 438.410 and 457.1260)
1.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of an appeal? If yes, please describe this process.
a.

What are the time frames defined for this process?

How does the MCP notify enrollees of any denials of a request for expedited resolution?

Have there been any complaints by Medicaid and CHIP enrollees that their requests for expedited appeals
have not been acted upon timely (e.g., within three working days). If yes, how many such complaints were
received in the year under review?

Record keeping and reporting requirements (42 C.F.R. §§ 438.416 and 457.1260)
1.

How are Medicaid and CHIP grievances and appeals registered and tracked for resolution? Is each
grievance and appeal tracked through to resolution?

How often is Medicaid and CHIP grievance and appeal information analyzed for trends?
a.

Who receives this analysis?

Does the MCP provide any information to the state relative to its grievances and appeals?

How long are Medicaid and CHIP grievance and appeal records retained?

To what extent is information on Medicaid and CHIP enrollee grievances and appeals analyzed and included
as part of your MCP’s Quality Assessment and Performance Improvement Program?

Continuation of benefits while the MCP or PIHP appeal and the state fair hearing are pending (42 C.F.R. §§
438.420)
1.

What happens to enrollee benefits once continuation of benefits has been denied by the MCP, and an
appeal has been filed by the enrollee or the treating physician?
a.

Are there any mechanisms in place to continue the benefits pending the outcome of the appeal? If yes,
under what circumstances?

184 | PROTOCOL THREE

Utilization Management Staff Interview
Instructions. MCP interview participants should include the Medical Director, utilization management directors or
managers, utilization management review staff, case managers or care coordinators, and any other individuals who
have information pertinent to these regulatory provisions. [Note: This interview can be combined with the Medical
Director interview or the Care Coordinators and Case Managers interview.]
The utilization management interview provides an opportunity to discuss with the MCP staff responsible for tracking
and managing the utilization of MCP services. Through these interviews, the reviewer(s) will assess delivery network,
service authorization; the use of practice guidelines, and grievances and appeals; and management of resources
across all MCP network provider sites where enrollees receive health care.
Availability of services (42 C.F.R. § 438.206)
1.

How frequently does the MCP evaluate the volume and enrollee access to LTSS services? What factors are
used in evaluation of the LTSS network?
a.

How does the MCP evaluate the expected utilization of institutional care in comparison with use of
home and community based services (HCBS) as an alternative?

How frequently does the MCP evaluate the volume of and enrollee access to family plan and women’s
health services? What factors are sued to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to specialist health services?
What factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to children’s dental care? What
factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to behavioral health services?
What factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to family planning and women’s
health services? What factors are used to evaluate the network?

How frequently does the MCP evaluate the volume of and enrollee access to any other specific services,
such as HIV and foster care services? What factors are used to evaluate the network?

Delivery network (42 C.F.R. §§ 438.206(b) and 457.1230(a))
1.

What procedures must a Medicaid and CHIP enrollee follow if he/she wishes to receive a second opinion?
a.

For what types of services are second opinions available?

Coverage and authorization of services (42 C.F.R. §§ 438.210 and 457.1230(d))
1.

What types of services require pre-authorization?

What are the MCP’s time frames for processing standard and expedited requests for service authorization?

How does the MCP monitor its compliance with these time frames?
a.

What sources of documentation exist to provide evidence of the monitoring by the MCP?

How often and under what circumstances are requesting providers consulted when the MCP makes service
authorization decisions?

PROTOCOL THREE | 185

To what extent does the MCP assess the consistency of authorization decisions? How does the MCP do
this?

What is the process when a decision is being made to deny authorization for a service?
a.

Who makes the decision to deny a request to authorize a service?

Please describe the process for notifying the requesting provider and the enrollee of any decision to deny,
limit, or discontinue authorization of services.
a.

What information is typically included in enrollee and provider notification?

What are the MCP’s time frames for notification?

To what extent if at all s inappropriate use of emergency rooms by your Medicaid and CHIP enrollees a
concern for your MCP?

Has your MCP investigated a potential relationship between inappropriate emergency department use and
enrollee access to routine and urgent care, or reviewed the most frequent diagnosis resulting in
inappropriate emergency department use?

10. What was the volume of denied claims for emergency and post-stabilization services in the most recent
year?
11. Does the authorization process differ between acute and primary services and LTSS, or any other
providers? If yes, how?
Grievance systems (42 C.F.R. § 438.228)
1.

What types of services require pre-authorization?

Please describe the process for notifying the requesting provider and the enrollee of any decision to deny,
limit, or discontinue authorization of services.
a.

What information is typically included in enrollee and provider notification?

What are the MCP’s time frames for notification?

How does your MCP track requests for covered services that the MCP or its providers has denied?

What was the volume of denied request for services in the most recent year?

Application of practice guidelines (42 C.F.R. §§ 438.236(c) and 457.1233(c))
1.

What practice guidelines have the MCP adopted?

To what extent are your utilization management review guidelines (criteria) consistent with these practice
guidelines?
a.

How do you promote or ensure consistency?

Please describe how utilization management review guidelines (criteria) are modified to reflect the adoption
or revision of practice guidelines.
a.

Are both sets of guidelines updated through the same process, at the same time?

186 | PROTOCOL THREE

Quality assessment and performance improvement program (42 C.F.R. §§ 438.330 and 457.1240(b))
1.

What information is analyzed to detect over- and under-utilization of services?
a.

Who is involved in the analysis and review of this information?

What, if any trends been identified?

What are the typical follow-up actions taken when either condition is discovered?

How does the MCP monitor LTSS utilization patterns? Are there any services for which specialized or
more focused utilization analysis is used?

Grievance system - General requirements (42 C.F.R. §§ 438.402 and 457.1260)
1.

Please describe the appeals process and the role of utilization management staff in the resolution process.
Elaborate on the particular steps, including time frames, in which utilization management staff is involved.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of an appeal? Please describe this process and its time frame.

Does the MCP’s grievance and appeal system differ for LTSS vs. acute and primary care services? If yes,
how?

Handling of grievances and appeals (42 C.F.R. §§ 438.406 and 457.1260)
1.

What MCP department or staff is responsible for assisting enrollees in using the MCP’s appeal or grievance
system, including completing forms, or taking other steps to resolve an appeal or grievance?

What are the qualifications and credentials of individuals who make decisions on grievances and appeals?
a.

How does the MCP ensure that these individuals have not been involved in any previous level of
review or decision-making?

How does the MCP ensure that these individuals have the appropriate clinical expertise in treating the
enrollee’s condition or disease, if deciding any of the following:
i.

An appeal of a denial that is based on lack of medical necessity

ii.

A grievance regarding denial of expedited resolution of an appeal

iii.

A grievance or appeal that involves clinical issues

Expedited resolution of appeals (42 C.F.R. §§ 438.410 and 457.1260)
1.

Is there a process in place for those instances when an enrollee’s health condition requires expedited
resolution of a grievance? If yes, please describe this process. What are the time frames defined for this
process?

How does the MCP notify enrollees of any denials of a request for expedited resolution?

Continuation of benefits while the MCP or PIHP appeal and the state Fair Hearing are pending (42 C.F.R. §
438.420)
1.

What happens to enrollee benefits once continuation of benefits has been denied by the MCP, and an
appeal has been filed by the enrollee or the treating physician or other provider, including providers of
LTSS?

PROTOCOL THREE | 187

Are there any mechanisms in place to continue the benefits pending the outcome of the appeal and if
so, under what circumstances?

How are enrollees notified of this mechanism?

188 | PROTOCOL THREE

Medical Directors Interview
Instructions. The interview with the Medical Director provides an opportunity to assess MCP processes for
authorizing services and coverage for those services. The interview will address such topics as provider involvement
in the review of criteria used in the utilization management process, consistency between utilization management
criteria and practice guidelines, and Quality Assessment and Performance Improvement efforts.
Coverage and authorization of services (42 C.F.R. §§ 438.210 and 457.1230(d))
1.

How does the MCP monitor its compliance with the state’s time frames for processing standard requests for
service authorization?

What are the MCP’s standards for processing expedited requests for service authorization? How does the
MCP monitor its compliance with these time frames?

Under what circumstances are requesting providers consulted when responding to service authorization
requests?

How does the MCP ensure consistent application of criteria used in making service authorization decisions?

What mechanism does the MCP use to assure that any decision to deny a service authorization request or
to authorize a service in an amount, duration or scope that is less than requested, be made by a health care
professional who has appropriate clinical expertise in treating the enrollees’ condition or disease or by a
professional with expertise in serving special populations (e.g. Developmental Disabilities), in special
services (e.g., Vocational Rehabilitation), or with other LTSS expertise as appropriate?

How are employees and any contractors used by the MCP to perform service authorization and utilization
management financially compensated?

Are they paid in any way other than on a straight salary or per case review basis?

Do their financial compensation arrangements involve the use of any financial incentives?

How does the MCP apply the definition of ‘medically necessary services’ to LTSS for activities that support
age-appropriate growth and development and/or the ability to attain, maintain or regain functional capacity?

Quality assessment and performance improvement program (42 C.F.R. §§ 438.330 and 457.1240(b))
1.

Does the MCP have any processes for reviewing claims, payment systems, encounter data, electronic
health records, and medical records to assess utilization of services?
a.

Does the MCP utilize a health information exchange process?

What reports on service utilization are regularly produced by these processes?

What are the most recent findings with respect to over- and under-utilization?

How does your MCP define enrollees with “special health care needs”? How are these enrollees identified
within your MCP?

How does your MCP assess the quality and appropriateness of care furnished to enrollees with special
health care needs? Please provide examples.

Does the state require your MCP to address a specific topic or topics in your performance improvement
projects? If yes, what types of projects are required? For each PIP, at a minimum, include how significant
improvement was measured, how improvement will be/was sustained, and how beneficiary health outcomes

PROTOCOL THREE | 189

and satisfaction will be/was measured, and how the intervention will/has improved access and/or quality of
care.

For duals-only MCPs, was a Medicare Advantage PIP substituted for a state-required PIP?

If CMS specifies any performance measures, what performance measures are collected and submitted,
if any?

Does the state require your MCP to evaluate the impact and effectiveness of its quality assessment and
performance improvement program?
a.

How does your MCP conduct its evaluation? What aspects of the program are included in the
evaluation?

How often does your MCP conduct its evaluation?

What were the findings of the MCP’s most recent self-evaluation?

What action did the MCP take as a result of these findings?

What is reported to the state, and how often?

What is your MCP’s role in the state’s evaluation?

What information, if any, does your MCP provide to the state?

What feedback, if any, does your MCP receive from the state? How does your MCP implement the
feedback?

190 | PROTOCOL THREE

Case Managers and Care Coordinators Interview
Instructions. Case managers and care coordinators typically are among the few MCP staff with opportunity to
interact closely and directly with Medicaid and CHIP enrollees. These individuals are often responsible for guiding
enrollees to the care and services available through their benefits and the provider network. These individuals play a
key role in assisting enrollees in managing and maintaining their health and managing complex conditions.
Interviewing these individuals will provide reviewers the opportunity to discuss topics surrounding MCP processes
related to service availability, enrollee needs and special populations, and continuity and coordination of care. [Note:
This interview can be combined with the Medical Director interview or the Utilization Management interview.]
Enrollee rights (42 C.F.R. §§ 438.100, and 42 C.F.R. § 438.206-210)
1.

How are the available options for LTSS identified and presented to enrollees?

How are enrollees engaged in decisions about the use of LTSS?

How is the enrollee’s right to be free from restraint or seclusion monitored for enrollees receiving LTSS?

Enrollee right to participate in decisions regarding his or her health care (42 C.F.R. §§ 438.100(b)(iv) and
457.1220)
1.

To what extent does the MCP allow enrollees to participate in care and treatment decisions? Please
describe some of the ways in which this is accomplished.

Availability of services (42 C.F.R. § 438.206)
1.
2.

How does the MCP evaluate the expected utilization of institutional care in comparison with the use of
HCBS as an alternative?
How does the MCP evaluate availability of services for individuals with intellectual and developmental
disabilities? What factors are used to evaluate the network?

How does the MCP evaluate availability of services for children with special health care needs? What factors
are used to evaluate the network?

How does the MCP evaluate availability of services for individuals with behavioral health conditions? What
factors are used to evaluate the network?

How does the MCP evaluate availability of services for dual-eligibles? What factors are used to evaluate the
network?

How does the MCP evaluate availability of services for individuals with HIV? What factors are used to
evaluate the network?

What methods does the MCP use to improve cultural competency?

Furnishing of services and timely access (42 C.F.R. §§ 438.206(c) and 457.1230(a))
1.

To what extent are services offered through the MCP available to Medicaid and CHIP enrollees and others
coordinating care 24 hours per day, 7 days per week when medically necessary?

What types of services require pre-authorization?

Coordination and continuity of care (42 C.F.R. §§ 438.208 and 457.1230(c))
1.

Does this MCP screen Medicaid and CHIP enrollees to identify those with special health care needs? If yes,
how is this implemented?

PROTOCOL THREE | 191

How are Medicaid and CHIP enrollees with special health care needs—including any individuals with special
health care needs identified by your MCP and any identified by the state Medicaid agency or its agent—
identified and tracked within your MCP?

Does this MCP assess Medicaid and CHIP enrollees with special health care needs? If yes, how are these
activities conducted?

Does this MCP require written treatment plans to be developed for enrollees with ongoing special conditions
that require a course of treatment or regular care monitoring? If yes, how is it decided which Medicaid and
CHIP enrollees will receive a written treatment plan?

If treatment plans are required by this MCP, how does the MCP ensure that treatment plans for individuals
with special health care needs address the needs identified by the assessment?

Please describe the treatment planning process for individuals with special health care needs and the
process for determining and assuring appropriate use of specialists.

Within the last year, how many treatment plans have been developed?

How many requests made by enrollees for review of treatment plans have been denied?

What were the reasons for these denials?

How many treatment plans have been denied?

What were the reasons for these denials?

What process(es) is/are used to coordinate services for enrollees?
a.

Are their different types of care coordination mechanisms for different types of enrollees? If yes, how
are these different and how do they work?

Who is responsible for coordinating the care of individuals with special health care needs?

10. What are the procedures for coordinating the services that the MCP furnishes to the enrollee with services
the Medicaid and CHIP enrollee receives from any other MCOs, PIHPs, and PAHPs?
11. If the MCP establishes separate coordination of care for medical services, MLTSS, and mental health and
substance abuse services, how does the MCP ensure exchange of necessary information between
providers?
12. How is post-acute care coordinated?
13. How are LTSS providers involved in person-centered assessment, person-centered care and service
planning, coordination and authorization processes?
Coverage and authorization (42 C.F.R. §§ 438.210 and 457.1230(d))
1.

What types of services require pre-authorization?

Are emergency back-up plans created for all enrollee’s? If not, how is the need for an emergency back-up
plan determined? How is the emergency back-up plan shared with all appropriate parties?

Quality assessment and performance improvement program (42 C.F.R. §§ 438.330 and 457.1240(b))
1.

What processes does the MCP have to detect underutilization and overutilization? What activities, such as
QAPI projects, has the MCP implemented to address these issues?

192 | PROTOCOL THREE

a.
2.

What are the results of these activities?

What activities, such as QAPI projects, has the MCP implemented to assess and improve care
coordination?
a.

What are the results of these activities?

PROTOCOL THREE | 193

Providers and Contractors Interview (as appropriate and time and resources permit)
Instructions. While interviewing providers and contractors requires additional time and resources, it is an opportunity
to obtain further information about MCP performance from those health care and LTSS professionals and institutions
that often serve as the first point of contact for Medicaid and CHIP members and health care providers. Provider and
contractor interviews should therefore be considered as an optional component of this protocol, to be considered
whenever there is a strong need for additional information and when time and resources permit. The interview
participants should be selected from the provider network and should offer representative view of the breadth of the
MCP’s primary care, specialist, LTSS, and institutional providers. These persons can often clarify issues pertaining to
communication, traversing the system, assuring enrollee rights, and delivery of care and services to the enrolled
population.
There are several ways to conduct the interview. It can be arranged with a group of individual health care
practitioners, a group of institution representatives, and a group of LTSS providers, or coordinated as one interview
for each group or as a combined group. Geographic location of providers should be considered, and conference calls
are a viable option for conducting an interview of this type, and often preferred by providers as only a brief
interruption in their daily activities. For this interview to be effective, reviewers should emphasize that this is an
opportunity to provide insight on the MCP’s performance and not an evaluation of the care and services offered to
Medicaid and CHIP enrollees.
Enrollee rights (42 C.F.R. §§ 438.10 and 457.1207) and Enrollee information (42 C.F.R. §§ 438.100 and
457.1220; and 42 C.F.R. §§ 438.206-210 and 457.1230(a-d))
1.

When the MCP’s enrollees present for services, do they appear to have a clear understanding of their rights,
responsibilities, and benefits? How to obtain services?

Does the MCP provide you with information on where to refer enrollees who are having difficulty
understanding the materials that have been provided to them by the MCP?

How often do you and your staff have to assist enrollees with understanding the materials provided by the
MCP?

Does the MCP require providers to have access to oral interpreter services?

Does the MCP provide your office with guidance or assistance is accessing interpreter services if
necessary?

Enrollee rights to receive information on available treatment options (42 C.F.R. §§ 438.102 and 457.1222; and
42 C.F.R. §§ 438.206-210 and 457.1230(a-d)); Provider-enrollee communications (42 C.F.R. §§ 438.100 and
457.1220; and 42 C.F.R. §§ 438.206-210 and 457.1230(a-d))
1.

Does the MCP place any limits on your ability to counsel or advise a Medicaid and CHIP enrollee on
treatment options that may be appropriate for the enrollee’s condition or disease?

Does the MCP encourage providers to share with enrollees information on available treatment options and
alternatives?
a.

Does this include options and alternatives that are within as well as those outside the scope of the
enrollees benefits? If yes, how does the MCP do this?

Availability of Services: Furnishing of services (42 C.F.R. §§ 438.206(c) and 457.1230(a))
1.

Are your hours of operation for Medicaid and CHIP enrollees different from the hours of operation for other
MCP enrollees? If yes, why?

Practice guidelines (42 C.F.R. §§ 438.236 and 457.1233(c))

194 | PROTOCOL THREE

Are affiliated providers/contractors consulted as practice guidelines are adopted and re-evaluated?

How does the MCP make providers/contractors aware of practice guidelines currently in use and those
under consideration for adoption?

Expedited resolution of appeals (42 C.F.R. §§ 438.410 and §457.1260)
1.

Have there been any instances in the most recent year under review when the MCP took any punitive action
against you for requesting an expedited resolution of an appeal on behalf of Medicaid and CHIP enrollees or
for supporting an enrollee’s appeal?

END OF WORKSHEETS FOR PROTOCOL 3

PROTOCOL THREE | 195

This page left blank for double-sided copying.

196 | PROTOCOL THREE

PROTOCOL 4. VALIDATION OF NETWORK
ADEQUACY
A MANDATORY EQR-RELATED ACTIVITY
RESERVED.

PROTOCOL FOUR
197

This page left blank for double-sided copying.

198 | PROTOCOL FOUR

PROTOCOL 5. VALIDATION OF ENCOUNTER
DATA REPORTED BY THE MEDICAID AND CHIP
MANAGED CARE PLAN
AN OPTIONAL EQR-RELATED ACTIVITY

ACTIVITY 2: REVIEW THE MCP’S CAPABILITY
ACTIVITY 3: ANALYZE ELECTRONIC ENCOUNTER DATA
ACTIVITY 4: REVIEW MEDICAL RECORDS
ACTIVITY 5: SUBMIT FINDINGS

BACKGROUND
Encounter data are the information related to the receipt of any item or
service by an enrollee in a managed care plan (MCP). It is often thought
of as the managed care equivalent of fee-for-service (FFS) claims.
Encounter data reflect that a provider rendered a specific service under a
managed care delivery system, regardless of if or how the MCP ultimately
reimbursed the provider. They contain substantially the same information
included on claim forms (e.g., UB-04 or CMS 1500), although not
necessarily in the same format. However, because some managed care
providers and/or services may be paid via capitation or episodes of care,
rather than based on a claim submitted for individual services rendered,
encounter data may be less complete or accurate than claim data. As
payment methodologies have begun to incorporate value-based payment
elements (such as bundled payment or episode payment), collecting
complete and accurate encounter data has become even more crucial.
Since 1999, CMS has required states to submit complete and accurate
enrollment and utilization data, including FFS claims and encounter
records, through the Medicaid Statistical Information System (MSIS). In
2011, CMS began working with state agencies and other stakeholders to
build a new data infrastructure to replace MSIS. The Transformed
Medicaid Statistical Information System (known as T-MSIS) is intended to
modernize the way states submit data about beneficiaries, providers,
MCPs, FFS claims, third-party liability, and encounters to CMS. States
must comply with the T-MSIS requirements and all associated guidance
for all managed care data submitted to CMS. 62

PROTOCOL FIVE

ACTIVITY 1: REVIEW STATE REQUIREMENTS

See August 10, 2018 letter to State Health Officials (SHO# 18-008) providing guidance to
states regarding expectations for Medicaid and CHIP data and ongoing T-MSIS implementation
at https://www.medicaid.gov/Federal-Policy-Guidance/downloads/SHO18008.pdf.

199

The availability of accurate and complete encounter data is important to the effective operation
and oversight of MCPs that serve enrollees covered by Medicaid and the Children’s Health
Insurance Program (CHIP) (See box, State Uses of Encounter Data).
Federal regulations at 42 C.F.R. § Part 438 include several provisions related to encounter data.
•

All providers must submit claims and/or encounters to
states for all services regardless of the method by which
a plan pays its providers (e.g., fee for services (FFS),
capitated, basis, or sub-capitation). (42 C.F.R. §
438.818(a))

State Uses of Encounter
Data
• Develop capitation rates
• Perform risk adjustment
• Measure quality

•

States must review and validate encounter data on
initial receipt from their MCPs, and again when they
submit it to CMS. (42 C.F.R. § 438.818(a)(2))

•

States must submit complete, accurate, and timely
encounter data to CMS in a standardized format (i.e.,
Transformed Medicaid Statistical Information System
(T-MSIS)). (42 C.F.R. § 438.818(a)(3))

•

CMS may impose penalties on states for noncompliance by withholding
Federal Financial Participation (FFP) funds. (42 C.F.R. § 438.818(c))

• Implement alternative payment
methods
• Conduct program integrity
• Engage in policy development

This protocol provides guidance to EQROs on validating the accuracy and
completeness of encounter data submitted by MCPs.

GETTING STARTED ON PROTOCOL 5
To complete this protocol, the EQRO undertakes five activities for each MCP (Figure 5.1).

200 | PROTOCOL FIVE

Figure 5.1. Protocol 5 Activities

Two supplemental resources are available to help EQROs validate encounter data:
•

Worksheets for Protocol 5. Encounter Data Tables, a set of worksheets that can be used to
document acceptable error rates and data element validity requirements, findings from the
review of individual encounter records, a comparison of findings to state-identified
benchmarks, results from the EQRO’s validation of medical records, and a suggested
format for reporting encounter data validation information in the EQR technical report.
Format for Reporting Encounter Data Validation Information in the EQR Technical Report

•

Appendix A. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems.

In addition, it may be helpful to refer to the CMS Encounter Data Toolkit, which contains
additional information and resources about the validation process. This toolkit is available at
https://www.medicaid.gov/medicaid-chip-program-information/by-topics/deliverysystems/downloads/medicaid-encounter-data-toolkit.pdf.
The remainder of this protocol outlines the steps associated with Activities 1 through 5.

PROTOCOL FIVE | 201

ACTIVITY 1: REVIEW STATE REQUIREMENTS
WORKSHEET 5.1
WORKSHEET 5.2

Activity 1 is intended to ensure the EQRO
has a complete understanding of a state’s
requirements for each MCP’s encounter
data (See box, Resources for Activity 1). At
the outset of Activity 1, the state should
provide the EQRO with at least the
following information:
1

Specific requirements regarding the
MCPs’ collection and submission of
encounters. Some states may
formalize these requirements in
contractual language or companion
guides. The state should provide the
EQRO with a detailed list of all
requirements, by plan and plan type

Resources for Activity 1
Before initiating Activity 1, EQROs should request all
available encounter data guidance from states, including
encounter reporting requirements and standards, data
dictionary, edit checks, and other documents.
Worksheet 5.1. Specification of Acceptable Error Rates
and Identified Areas of Concern
• Provides guidance for the EQRO’s review of a state’s
specific requirements for reporting encounters
Worksheet 5.2. Data Element Validity Requirements
• Template for the EQRO to document the state’s
specific requirements for validating each data element
by type of service

Requirements regarding the types of encounters that must be validated (e.g., inpatient
hospital, professional, home health). The state may find it difficult to integrate some types of
encounters (e.g., non-emergency transportation or atypical providers) into its data systems.
Whenever possible, the state should direct the EQRO to alternative sources to validate this
information. Note that under this protocol, a state could direct the EQRO to validate all of an
MCP’s encounter data or a subset of an MCP’s encounters. If the state chooses to validate
a subset of the encounter data based on provider type, it must validate all encounters for
the selected provider type across all MCPs. If the state chooses to validate encounter data
for a subset of MCPs, it must validate all encounters for the subset of MCPs

Standards for the submitted data, including the following:
○

An operational definition of an “encounter,” such as adjudication status, and other
relevant details

○

Types of encounters MCPs must report (e.g., inpatient hospital, outpatient,
professional, home health)

○

Format in which encounters must be submitted (837 standard transaction, proprietary)

202 | PROTOCOL FIVE

○

Objective standards to which encounter data will be compared (e.g., number of
beneficiaries with at least one encounter)

State standards for encounter data completeness and accuracy. The state should
clearly specify acceptable rates of accuracy and completeness for each data element for
each field for each encounter type, which may depend on the intended use of the encounter
data. Although initial error rates may be higher, each MCP’s targeted error rate should be
below 5 percent for each time period examined. The state should align its own standards
with those required to satisfy T-MSIS requirements

Data dictionary and companion guides. States often cite data dictionaries or companion
guides in managed care contract language for reference to accountability and standards for
encounter data. Those may be updated on a more regular basis than the contracts
themselves. For states that employ a fiscal intermediary, the intermediary may be the best
source of this information

Description of the information flow from the MCP to the state, including the role of any
contractors or data intermediaries. States that use separate organizations for medical and
behavioral health should include details about how the data are collected and integrated
into a single system, as well as challenges the EQRO may face in handling these data

A list and description of automated edits or checks performed on the data when
received into the state system (Medicaid Management Information System or data
warehouse). This should include information about how the system handles encounters that
fail an edit check. For example, does the system reject an entire file if one encounter is
rejected?

The timeliness requirements for data submissions (e.g., how far from the original date
of service the record must be submitted), and standards for timeliness, as applicable and as
laid out by the state in contract documents. States are increasingly able to process high
volumes of records on a daily basis, while some prefer a monthly submission from plans.
States also may have various tolerance levels for what percentage of records must meet
particular timeliness standards

Any EQR validation reports from previous years. Previous reports can provide useful
data points for determining how much progress MCPs have made in improving data quality
and completeness, as well as giving a state picture of improvement or challenges over time

10 Any other information relevant to encounter data validation. States may find they use
other documentation or context in their own analyses of their MCP’s encounter data. If
supplementary information will provide relevant context to encounter validation, such as a
list of excluded providers, it should be provided to the EQRO

ACTIVITY 2: REVIEW THE MCP’S CAPABILITY
Activity 2 is intended to evaluate an MCP’s ability to collect complete and accurate encounter
data. Before assessing the output produced by the MCP’s information system, the EQRO should
determine whether the system is able to collect and report high quality encounter data. To do so,
the EQRO should assess the information system in two steps (described in more detail below):
1

Review the MCP’s most recently completed Information System Capacity Assessment (ISCA)

Interview MCP personnel to clarify ISCA findings as necessary

PROTOCOL FIVE | 203

Step 1: Review the MCP’s ISCA

WORKSHEET A.1
WORKSHEET A.2
WORKSHEET 5.1

The EQRO should determine whether the MCP has
completed an ISCA review within the past two
years. 63 If a recent ISCA has been completed, the
EQRO should review the findings. If the MCP has
not conducted an ISCA within the previous two
years, the EQRO must conduct one consistent with
the processes discussed in Appendix A.
The EQRO should review the MCP’s ISCA to
identify weaknesses in the MCP’s information
systems (See box, Resources for Activity 2, Step 1).
This assessment determines where and how
information systems may be vulnerable to
incomplete or inaccurate data capture or
processing, integration, storage, or reporting. Based
on the findings from the ISCA, the EQRO should
understand the following:

Resources for Activity 2, Step 1
Worksheet 5.1. Specification of Acceptable
Error Rates and Identified Areas of Concern
• Provides guidance for the EQRO’s review
of a state’s specific requirements for
reporting encounters
Appendix A guides the information systems
review. The appendix includes two
worksheets:
• ISCA Worksheet A.1 is the tool used for the
assessment
• ISCA Worksheet A.2 is used by the EQRO
to assess the adequacy of MCP policies
and procedures based on the information
collected in Worksheet A.1

IT system architecture, file structure, information flow, and data processing procedures

Specific programming language used by the system (e.g., SQL)

Process by which the MCP modifies its source code to address changes in state reporting
requirements (Note: The EQRO should obtain all source code from the MCP)

Other claims/encounter processing issues

○

How the system handles voids, adjustments, crossovers, and records not requiring
payments, such as for sub-capitated arrangements

○

Whether the system verifies encounters at both header and detail levels 64

○

Whether there are processes in place to identify “orphan” header or detail records 65

Completeness of data
○

Whether there any service types (e.g., non-emergency transportation or behavioral
health) not in the system

There is no statutory or regulatory requirement for the frequency with which ISCAs should be conducted. Each state must
determine the maximum interval between assessments of MCP information systems, balancing the cost to the state and
burden on the MCP with the need to ensure that changes to the MCP’s information systems are assessed frequently enough
to support accurate performance measurement.
64
The detail-level on an encounter refers to information included on the individual lines contained within the encounter (such
as charges or procedure codes for multiple services provided within a single visit); the header-level refers to information
provided at the claim level (such as beneficiary ID, provider ID, date of service, and diagnoses).
63

An orphan encounter is one in which there are one or more detail records without an associated header record, or vice versa.

204 | PROTOCOL FIVE

Written policies and procedures for edits and audits

Claims/encounter system demonstration
○

Whether the system permits working with the data in a “test” environment

Processes for merging and/or transferring data

Processes for encounter data intake, logging, adjudication, and denial

10 Audits performed to assure data quality and accuracy and processing timeliness
11 Maintenance and updating of provider data, including how the MCP identifies providers or
organizations excluded from the Medicaid program each month (e.g., List of Excluded
Individuals and Entities), and whether the MCP requires its provider network to update
provider data each month
12 Processing of enrollment data, including a description of how the system identifies
beneficiaries as information changes over time (e.g., how the system handles name and
address changes)
13 Specific claims and encounter verification procedures
14 Frequency of information updates (e.g., how often does the MCP update its provider table?)
15 Management of enrollment and disenrollment information
During the review of ISCA findings, if
the EQRO identifies issues that may
contribute to inaccurate or incomplete
encounter data, the EQRO should list
any concerns about the encounter data
in Column 4 of Worksheet 5.1.
Specification of Acceptable Error Rates
and Identified Areas of Concern for
each encounter type listed (See box,
Potential Causes of Encounter Data
Errors by MCPs).

Potential Causes of Encounter Data Errors by MCPs
• Non-standard codes or forms
• Inadequate front-end data edits
• Lack of provider contractual requirements that tie payment to
data submission
• Use of default dates of service or provider identifiers
• Failure to collect key demographic data elements
• Out of date or incomplete reference tables
• Failure to collect Medicare crossover claims
• Inconsistent use of adjusted and void claims

Step 2: Interview MCP Personnel

After reviewing the findings from the ISCA, the EQRO should conduct follow-up interviews with
MCP personnel as needed to supplement the information in the ISCA and ensure its
understanding of the MCP’s information systems and processes. The EQRO should refer to
ISCA components that the MCP uses to produce performance measures, including enrollment,
medical, pharmacy, provider, lab, and other ancillary or supplemental data sources.

ACTIVITY 3: ANALYZE ELECTRONIC ENCOUNTER DATA
Activity 3 is the core function used to determine the validity of the encounter data. When the
EQRO has completed the steps within this activity, it should know whether the data are
complete, of high quality, and can be used for analysis of quality, access, program integrity
monitoring, among other critical state activities. If the EQRO cannot confirm the quality of the
data after completing this activity, it should not proceed to Activity 4, the Medical Record
Review. Instead, the EQRO should work closely with the state or plans to determine underlying

PROTOCOL FIVE | 205

problems or acquire additional information to determine the quality and usefulness of the data
submitted. Difficulties completing this analysis may need to be summarized for the state as
indicating serious data quality issues.
The EQRO should use the information obtained from these analyses, the ISCA tool, follow-up
interviews, and the results of state edits to assess the completeness and accuracy of the MCP’s
encounter data. The results of Activity 3 will inform the development of a long-term monitoring
strategy for assessing the quality of the encounter data. As the data evolve over time, the
EQRO will be able to design targeted validation strategies to identify problem areas requiring
resource intensive medical record review.
Under this activity, the EQRO should carry out the following Steps 1 through 4.
Step 1: Develop a Data Quality Test Plan Based On Data Element Validity Requirements

The EQRO should use the information obtained from Activities 1 and 2 (including the ISCA
review and follow-up interviews with MCP staff) to develop a data quality test plan. The plan
should:
•

Account for front-end edits already built into the MCP’s data system so that it focuses on
issues that the MCP may have inadvertently missed or allowed for other reasons

•

Specify the areas to be tested and the expected results

To be of greatest use to states and other stakeholders, the EQRO should develop a plan that
addresses the following questions:
1

The general magnitude of missing encounter data. The EQRO should use information
from the MCP about encounters that fail front-end edits and the reasons for these failures to
determine whether, and how much, encounter data is missing. The EQRO should compare
these results with normative data on encounters for similar populations for this purpose.
Examples of the use of benchmarks for assessing encounter data completeness are
available in the Encounter Data Toolkit, available at https://www.medicaid.gov/medicaidchip-program-information/by-topics/delivery-systems/downloads/medicaid-encounter-datatoolkit.pdf

Types of encounters that may be missing. MCPs that pay for “bundled” services (e.g.,
prenatal care) or that capitate providers (e.g., for primary care) may not receive complete
encounter information from these providers. The EQRO should apply specific knowledge
about the MCP’s contractual relationships with providers to identify specific areas to look for
missing services. The EQRO should obtain information from the MCP on the use of
bundled payment and capitation to inform its plan

Overall data quality issues. The EQRO should identify specific data quality problems such
as inability to process or retain certain fields, or limited coding specificity on the encounter
data record

MCP data submission issues. The EQRO should identify problems the MCP has
compiling its encounter data and submitting the data files to the state

206 | PROTOCOL FIVE

Step 2: Encounter Data Macro-Analysis—Verification of Data Integrity

WORKSHEET 5.3

Steps 2 and 3 of Activity 3 are closely related. When the
EQRO reviews the data for accuracy and completeness,
it conducts both macro- and micro-analyses. Step 2
describes the macro-analysis, while Step 3 describes the
micro-analysis.
In Step 2, the EQRO should:
•

Analyze and interpret data in specific fields

•

Check the data for volume and consistency

Resources for Activity 3, Step 2
Worksheet 5.3. Evaluation of Submitted
Fields
• Template for the EQRO to document
its findings for each data element

Without duplicating the state’s edit checks, the EQRO should analyze and interpret data in
submitted fields. In addition:
1

Is there information in the field, and is that information of the type requested?
○

The EQRO should check each field to determine whether its values are of the type and
size found in the state’s data dictionary, or in nationally recognized standards. For
example, if CPT®-4 codes are requested, the field should have five digits. If the state’s
Medicaid/CHIP beneficiary ID is requested, the field should contain the correct number
of letters and characters

Are the values valid and in the correct format?
○

To what extent are the values in the field valid? For instance, if ICD-10 diagnosis codes
have been requested, are the values in the diagnosis field valid for that standard?

○

Do critical fields contain non-missing values in the correct format and specificity (e.g.,
maximum number of characters in a diagnosis) and that values are consistent across fields?

Are the data available?
○

Are all required data elements reported?

○

Do the data exist for all service types?

○

When viewed by date of service, are there gaps in the data?

Do the data meet basic consistency expectations?
○

Is beneficiary enrollment consistent over time?

○

Are the number of encounters consistent over time?

○

When broken out by population subgroups or service types, does consistency persist?

Are the state’s identifiers (IDs) accurately incorporated into the MCP’s information system?
○

The EQRO should compare the encounter data file to the state’s eligibility file and
check for accuracy of the IDs and other eligibility information (e.g., age, sex, and
eligibility category). In addition, the EQRO should determine whether there are
encounter data for the expected proportion of beneficiaries in comparison to utilization
norms for similar populations
PROTOCOL FIVE | 207

Is the information for each critical field within required ranges and is the volume of data
consistent with the MCP’s enrollment? For example, can the following types of questions be
answered with the data:
○

What is the rate of emergency department utilization per 1,000 member months?

○

What percentage of beneficiaries have at least one encounter during the year?

Note: The EQRO should automate these analyses and perform them as a standard data review
process. The EQRO should perform these analyses for each service type (e.g., inpatient
hospital, outpatient, professional, home health, durable medical equipment) and for each data
field within a service type.
Step 3: Encounter Data Micro-Analysis—Generate and Review Analytic Reports

WORKSHEET 5.3

In Step 3, the EQRO should move beyond analyses
focused on data integrity and that are field-specific to
analyses that cross fields and provide a broader view of
whether the data can be used for meaningful analyses.
Often data elements may meet basic expectations, but
until multiple fields are used together for analysis, some
data quality issues may not be detected. Examples of
analytic reports that can detect broader data quality
issues are:
•

Resources for Activity 3, Step 3
Worksheet 5.3. Evaluation of Submitted
Fields
• Template for the EQRO to document
its findings for each data element

Reasonability tests
○

The EQRO should develop frequency distributions of the values and compare them to
normative data from similar populations to determine whether the values make sense.
For example,

○

If “place-of-service” is a required field, the values should be distributed across a range
of values (e.g., IP hospital, OP hospital, ED, or office)

○

The number of enrollees, the number of encounters, and counts and totals for various
eligibility categories or demographic subgroups, diagnoses, or types of service

○

Frequency distributions on specific fields, as well as on the variables created explicitly
for data validation purposes (e.g., beneficiary age from date of birth)

○

Distributions on subsets of data, especially where there are specific concerns about
data validity. For example, if the EQRO finds a low rate of utilization for outpatient
services, it could analyze the data by provider zip code to determine whether the files
are missing specific zip codes, causing the system to reject records. By taking a
deeper dive into the data, the EQRO could detect a different problem than originally
expected

○

Univariate statistics (e.g., means, medians, quartiles, and modes) as appropriate.
The EQRO should check the output of these reports for reasonableness and to detect
specific problems such as entire categories of data missing from the regular data
submissions

208 | PROTOCOL FIVE

•

Analyses by dates of service versus adjudication dates
○

•

Checks by provider types
○

•

The EQRO should review the data by provider type to identify missing provider types
and examine fluctuations in patient visits by provider type for specified time periods.
The EQRO should compare the distribution of encounters by provider type to
normative information. The EQRO should also examine diagnosis or procedure codes
by provider type to ensure that the relationship between provider specialty and the
services rendered is consistent

Relational analyses by service type or episodes of care
○

The relationship between ancillary services (e.g., labs, x-rays, etc.) and visits

○

The relationship between outpatient visits and the number of prescriptions dispensed

○

The relationship between primary and specialty care visits

○

Outpatient services associated with inpatient admissions

○

Grouped services expected in particular types of visit or episodes of care

○

Other relationships between service types previously identified as problematic through
the ISCA, front-end edits, or other EQRO validation activities.

Analyses broken out by demographic group or subpopulation
○

•

Analyzing the data by dates of service and by adjudication dates can detect issues in
consistency over time. Inconsistent processing can indicate other problems within the
MCP’s IS system, which may impact data validity. After establishing the length of time
between service and adjudication dates, the EQRO should compare them with
standards or benchmarks for data submission and processing

If not addressed already in the MCP’s front-end edits, the EQRO should conduct
analyses that take a beneficiary’s age and gender into account. For example, the
EQRO could verify that a gender-specific diagnosis (e.g., endometriosis) or procedure
(e.g., caesarean delivery) is consistent with the beneficiary’s age and gender derived
from the encounter header record or the beneficiary’s enrollment record

Analytic questions
○

The EQRO should use information gathered in previous steps to select a question or
series of questions it might answer using the encounter data as another step in
determining quality and usability. For instance, the EQRO could take a particular
measure of interest to the state and replicate it across all MCPs or within MCPs, such
as the number of beneficiaries per primary care provider

The EQRO should conduct these analyses on the encounter data and compare the results to
external benchmark information (Step 4). As part of the review, the EQRO should display the
data quality findings graphically to identify issues for further investigation and to communicate
the results of the data quality review. The EQRO should generate these reports for each MCP
and on the entire encounter data set for all MCPs to account for problems associated with small
numbers of encounters for individual MCPs. For examples of these types of displays, see
Section 5 of the CMS Encounter Data Toolkit, available at https://www.medicaid.gov/medicaidchip-program-information/by-topics/delivery-systems/downloads/medicaid-encounter-datatoolkit.pdf.
PROTOCOL FIVE | 209

Step 4: Compare Findings to State-Identified Benchmarks

WORKSHEET 5.4

In this step, the EQRO compares the encounter data
submitted by each MCP to benchmarks identified by the
state. The EQRO will need to identify and document
these benchmarks. The benchmarks can be obtained
from various sources, including:

Resources for Activity 3, Step 4
Worksheet 5.4. Benchmark Utilization
Rates
• Template for the EQRO to compare
findings to state-identified
benchmarks

•

Aggregate encounter data from all Medicaid or CHIP
MCPs in the state

•

Historical FFS or PCCM data

•

Other comparable states

•

Encounter data toolkit, available at https://www.medicaid.gov/medicaid-chip-programinformation/by-topics/delivery-systems/downloads/medicaid-encounter-data-toolkit.pdf

•

Other benchmarks, such as MCP financial reports, commercial MCPs, national standards,
HEDIS®, or the Child and Adult Core Set measures 66

The EQRO should understand which differences from comparison data require further
investigation. For example, emergency room utilization might be lower in managed care than in
FFS. However, large swings in utilization from one time period to the next, or differences from
the benchmark that are not explained by delivery system differences may indicate incomplete or
erroneous encounter data, rather than a difference in provider practice patterns. The EQRO
should vet its assumptions about changes in utilization with the MCPs and the state to
determine what follow-up analyses might be required. For example, unusual changes in
utilization and outcomes may occur after a natural disaster (such as a hurricane). The EQRO
should discuss anomalous findings with the state to assess underlying factors that may
utilization or outcomes.

ACTIVITY 4: REVIEW MEDICAL RECORDS
WORKSHEET 5.5
WORKSHEET 5.6

The purpose of Activity 4 is to confirm the findings from the analysis of encounter data
performed in Activity 3, using retrospective reviews of patient medical records. This activity
makes the following assumptions about the record review:

HEDIS® benchmarks are published annually by NCQA in The State of Healthcare Quality Report, available at
http://www.ncqa.org/report-cards/health-plans/state-of-health-care-quality. Child and Adult Core Set benchmarks are
available at https://www.medicaid.gov/medicaid/quality-of-care/index.html.

210 | PROTOCOL FIVE

Reviews performed under the guidance of this
protocol and activity should be independent of
record reviews performed for all other purposes,
including those performed to validate performance
measures, for program integrity, etc.

The state will determine the timing and frequency of
all medical record reviews

Once the state has determined a review of medical
records is appropriate, the EQRO will draw a sample
of records for validation on a regular and periodic
basis, as directed by the state agency

EQROs should approach the validation of encounters
from medical records as if they are research questions
with clear hypotheses, well-defined sampling
methodology, and predetermined error tolerances.
Questions under consideration for medical record review
generally fall into the following categories:
1

Resources for Activity 4
Worksheet 5.5. Medical Record Review
for Encounter Data Validation
• The Event Validation Table indicates
whether an encounter record
matches the medical record, and vice
versa
• The Data Field Validation Table
indicates whether codes or other data
fields in the encounter record match
the medical record, and vice versa
Worksheet 5.6. Medical Record Review
Results Summary Sheet
• Summarizes the results of the
medical record review, including the
error rate and reasons for errors

Questions of Description
○

Are all of the diagnosis codes in the patient’s medical record on the associated
encounter?

○

Are all of the procedure codes in the patient’s medical record on the associated
encounter?

○

Does the Date of Birth (DOB) listed in the beneficiary’s medical record match the DOB
found on the encounter header?

Questions of Relationship
○

Are there differences in the number of diagnoses reported for women compared to
men?

○

Are there differences in the distribution of Evaluation & Management (E&M) procedure
codes by age group?

○

Are there differences in the utilization of specific procedure codes by geographic area
(e.g., county)?

Questions of Comparison
○

Are there differences in the average number of diagnoses coded on the encounter
records compared to those found on FFS or PCCM claims?

○

Are there differences in the distribution of E&M procedure codes (i.e., 99201 – 99205)
on the encounter records compared to those found on FFS or PCCM claims?

○

Are there differences in the distribution of Place of Service codes on the encounter
records compared to those found on FFS or PCCM claims?

PROTOCOL FIVE | 211

EQROs should limit each medical record review to a specific encounter type (e.g., inpatient
hospital admissions, physician office visits). The EQRO should ensure that in narrowing the
scope of the review, it does not overlook service types that are vulnerable to undercounting
(such as prenatal and postpartum visits).
EQROs should determine the sample size for the medical record review using standard
sampling methodology (See box, Sampling Guidance for Medical Record Review). The sample
size will depend in part on the minimum error rate the EQRO must detect and the number of
subpopulations for which validation is conducted. Note that it is not appropriate to substitute a
record that is missing. Substitution may be allowed if a medical record is out of the office for
legal review.
Sampling Guidance for Medical Record Review
• See Appendix B for an overview of sampling approaches and guidance for calculating sample sizes
• Set sample sizes for medical record review sufficient to estimate the error rate for each type of encounter within
each population, with equal precision for each time period under review
• It may be appropriate to allow the substitution of a medical record if it is out of the office for legal review. However,
it is not appropriate to substitute a record that is missing
• A statistician or other staff with expertise in sample design and implementation should advise the state and/or
EQRO on the appropriate sampling strategy for the medical record review

Once the sample of medical records is selected, the EQRO needs to request the medical
records from providers, compare the content of the encounter records and medical records, and
document findings. The state should provide written guidance to the EQRO about the
procedures for conducting the medical record review, including the reporting requirements, the
data elements chosen for validation, and the error categories used. The EQRO should employ
experienced clinical coders to review codes based on the diagnoses stated by the provider in
the patient’s medical record.
To obtain medical records for review, the EQRO should give each provider a list with each
patient’s name, age, and sex, the provider’s name, and the target dates of service. This
information should be sufficient for the provider to identify the beneficiary and locate the correct
record
Guidelines should describe exactly how to document the findings of medical record review and
should include:
•

Directions for reviewing medical records

•

Instructions for evaluating conflicting documents

•

Instructions on what to do when no code can be readily assigned

•

Use of optional codes

•

Definitions of what constitutes an “error”

•

Lists and locations of approved reference materials

•

Whom to consult for additional assistance

212 | PROTOCOL FIVE

In defining what constitutes an error, the state should consider the following:
•

Designate certain errors as “critical” depending on the intended use of the data. These
designations may evolve over time as encounter data issues change. For example, the
initial stages of analysis may focus on diagnosis (ICD-10) and procedure (CPT®) codes
rather than provider specialty or place of service codes. The latter two fields may be of little
value if the former fields are inaccurate

•

Distinguish error “tiers” (e.g., critical, serious, moderate), which may permit use of
encounter data that may be incomplete or have some inaccuracies

ACTIVITY 5: SUBMIT FINDINGS
WORKSHEET 5.7

After the completion of Activities 1 through 4, the EQRO
Resources for Activity 5
should create data tables that display summary statistics
for the information obtained from each MCP. Summarizing
Worksheet 5.7. Suggested Format for
the information in tables makes it easier to evaluate the
Reporting Encounter Data Validation
Information in the EQR Technical
findings and highlight patterns in the accuracy and
Report
completeness of the data. The EQRO should draft a
narrative to accompany the tables, highlighting individual
MCP issues and providing recommendations to each MCP
and the state about improving the quality of the encounter data.
In its findings and recommendations, the EQRO should assess the MCP’s ability to provide the
state with encounter data that meets the quality standards for submission to the state for use in
T-MSIS. The EQRO should also assess the MCP’s ability to produce reliable and valid
performance measures as specified in the managed care quality strategy.
END OF PROTOCOL 5

PROTOCOL FIVE | 213

WORKSHEETS FOR PROTOCOL 5:
ENCOUNTER DATA TABLES
Instructions. Use these or similar worksheets as a guide when validating encounter data. The encounter data
worksheets 5.1 through 5.4 are intended to help document acceptable error rates and data element validity
requirements, findings from the review of individual encounter records, and comparison of findings to state-identified
benchmarks. Worksheet 5.5. Medical Record Review for Encounter Data Validation is intended to help record results
from the EQRO’s validation of medical records. Worksheet 5.6. Medical Record Review Results Summary Sheet
summarizes the results of the medical record review, including the error rate and reasons for errors.
Worksheet name

Protocol activity and step

Worksheet 5.1. Specification of Acceptable Error Rates
and Identified Areas of Concern
Worksheet 5.2. Data Element Validity Requirements

Activity 1. Step 1. Review the MCPs’ ISCA

Worksheet 5.1. Specification of Acceptable Error Rates
and Identified Areas of Concern
ISCA Worksheet A.1. ISCA Tool
ISCA Worksheet A.2. ISCA Worksheet & Interview
Guide

Activity 2. Step 1. Review the MCP’s Capability

Worksheet 5.3 Evaluation of Submitted Fields
Worksheet 5.4. Benchmark Utilization Rates

Activity 3. Analyze Electronic Encounter Data

Worksheet 5.5. Medical Record Review for Encounter
Data Validation
Worksheet 5.6. Medical Record Review Summary Sheet

Activity 4. Step 1. Obtain and Review Medical Records
and Document Findings

Worksheet 5.7. Suggested Format for Reporting
Encounter Data Validation Information in the EQR
Technical Report

Activity 5. Submit Findings

214 | PROTOCOL FIVE

Worksheet 5.1. Specification of Acceptable Error Rates and Identified Areas of Concern
Instructions. Worksheet 5.1 provides guidance for the EQRO’s review of a state’s specific requirements for reporting
encounters. The EQRO should add rows as necessary to include all service types used by the state. Definitions for
this activity are as follows:
•

Encounter Data Error Types
•

Missing. A service rendered for which there is no encounter record

•

Surplus. An encounter submitted for a service that was never rendered, or which duplicates another record

•

Erroneous. Services rendered where there is an error in the encounter record

•

Acceptable Error Rate. For each type of service (e.g., inpatient) and error (e.g., missing), the EQRO should
document the state’s acceptable error rate. In Worksheet 5.1, the acceptable error rate column expresses this
rate as the percentage of missing, surplus, or erroneous records the state will accept from the MCP. Files with an
error rate exceeding the thresholds are unacceptable. For example, a state might set error thresholds for office
visits at less than 10 percent for missing encounters, less than 2 percent for surplus encounters, and less than 5
percent for encounters with erroneous information. If the state expresses its error tolerance in a different way, the
EQRO should adjust the acceptable error rate accordingly.

•

Areas of Concern. Based on the ISCA and other information, the EQRO should identify errors that it reasonably
expects might occur. The EQRO could derive this information from work it performs in Protocol 2, Validation of
Performance Measures Reported by the MCP. It should use the information to guide subsequent reviews.
Service type

Acceptable
error rate

Error type

Office visit – includes all services,
except dental and mental health /
substance abuse

Missing
Surplus
Erroneous

<
<
<

%
%
%

Office visit – includes mental health /
substance abuse services only

Missing
Surplus
Erroneous

<
<
<

%
%
%

Office visit – includes dental services
only

Missing
Surplus
Erroneous

<
<
<

%
%
%

Inpatient admission – includes all IP
services, except mental health /
substance abuse services

Missing
Surplus
Erroneous

<
<
<

%
%
%

Inpatient admission – includes
mental health / substance abuse
services only

Missing
Surplus
Erroneous

<
<
<

%
%
%

Other types of encounters (e.g.,
emergency department, lab / x-ray,
pharmacy, physical therapy)

Missing
Surplus
Erroneous

<
<
<

%
%
%

Area of concern (Yes/No/Describe)

PROTOCOL FIVE | 215

Worksheet 5.2. Data Element Validity Requirements
Instructions. The EQRO should document clearly the state’s specific requirements for validating each data element
by type of service. The EQRO should then evaluate each file received to validate the specific data elements. The
EQRO should add rows as necessary to incorporate all data elements for which the state identifies specific validation
requirements. Definitions for this activity include:
•

Expectation. The EQRO should use this column to describe the general requirement(s) for validating each data
element.

•

Validation Criteria. The EQRO should use this column to document the validation threshold for each data
element. Typically, the column will include a quantitative expression of the description in the Expectation column.
Examples are shown in Worksheet 5.2.
Service type:
Data element

Expectation

Examples of validation criteria

Enrollee ID

A valid member ID (e.g., Medicaid ID
Number; SSN) as documented in the
state’s eligibility file.

98% valid.

Date of service (FDOS;
LDOS)

Dates of service should be distributed
across the entire period analyzed. Look for
large month-to-month increases or
decreases. Also, look for months with
encounters that may be missing entirely.

Calculate the average number of
encounters per month over the
period specified in the study. In
general, month-to-month differences
should be relatively small. Document
any outliers and request an
explanation from the MCP.

Unit of service (Quantity)

This field should generally include the
units billed for each type of medical
service (e.g., 2 units ≥ 23 minutes through
37 minutes).

X% non-zero.

Procedure code

Should include valid CPT® and HCPCS
values, or another state-approved code.

At least 98% of the values in this
field should be valid (i.e., non-zero,
not blank, and not 8-or-filled) and in
the expected format.

Revenue code (Hospital)

If the facility uses a UB04 claim form, this
field should always be populated on
inpatient encounters.

At least 98% of the values for this
field on inpatient claims should be
valid and in the expected format.

216 | PROTOCOL FIVE

< Y% should be 1 if CPT® code in
range 99200-99215, 99241-99291.

Worksheet 5.3. Evaluation of Submitted Fields
Instructions. As the EQRO reviews encounter records, it should document its findings for each data element on
Worksheet 5.3 or a similar form. To complete Worksheet 5.3 or similar tool, ask the following questions:
1. Is there information in the field, and is the information of the type (e.g., numeric) and format (e.g., MM/DD/YYYY)
required?
•

The EQRO should check each data element to determine whether its values are of the type and size specified
in the state’s data dictionary. For example, if CPT®-4 codes are required, the field should have 5 characters. If
the state’s Medicaid/CHIP beneficiary ID is required, the field should include the specified number of alpha,
numeric, or alphanumeric characters.

2. Compared to a generally accepted external standard, are the values in the specified field valid? For example, do
the values in the field PROC-CODE match those found in the ICD-10-CM tables?
Field is populated
Required field

Correct type
#

Correct size
#

Value is valid
#

Member ID
Plan ID
Billing provider
ID
Rendering
provider ID
Primary
diagnosis code
Primary
procedure code
First date of
service
Last date of
service
Quantity (units)
Add rows as
needed

PROTOCOL FIVE | 217

Worksheet 5.4. Benchmark Utilization Rates
Instructions. The EQRO should use this worksheet to compare its findings to state-identified benchmarks. Revise
the column headings to reflect the specific benchmarks identified by the state. EQROs should add measures as
specified by state validation requirements. CMS suggests that eligibility measures, if included, should align with the
eligibility group code in T-MSIS.

Measure
Inpatient discharges
Inpatient LOS
Overall
By high-volume MS-DRGs
By eligibility category/cohort
Ambulatory surgeries
Total number of surgeries
By high-volume CPTs® or ambulatory
surgery categories
Total surgeries (per 1,000 members)
By high-volume CPTs® or ambulatory
surgery categories
Providers
Primary care physicians
Specialists
Other (e.g., mental health providers)
Enrollees
Total number of enrollees
By eligibility category
By age, gender categories
Service utilization
Total number of service users
By eligibility category
By age, gender categories
Visits
Total number of visits
Average visits per enrollee
Average visits per user
By visit type (e.g., well-child)
Other service types (e.g., Rx)
Total number by service type
Encounters by enrollee/service type
Encounters by enrollee/service type

218 | PROTOCOL FIVE

MCP rate

FFS/PCCM
rate

Comparable
state(s) rate(s)

Other
comparable
rate (specify)

Worksheet 5.5. Medical Record Review for Encounter Data Validation
Instructions. Complete Worksheet 5.5 for each record in the sample. Transfer results from the Event Validation and
the Data Field Validation tables to Worksheet 5.6. Medical Record Review Results Summary Sheet.
Reviewer name:

Review completion date:

Data element

Field value

Data element

Field value

Medical record ID number

Practice name

Patient name

Practice TIN

Patient ID number

Rendering provider name

Patient gender

Rendering provider PIN

Patient date of birth

Primary diagnosis

First date of service

Principal procedure

Last date of service

Event Validation Table

Line number

Procedure

Event noted on
encounter
record

Event noted in
medical record

Match

No match

1
…
…
…
N
Note: Include one line for each procedure in the record for selected date.
If no match is found (i.e., the event is missing from either the medical record or the encounter record), record the
results on the Medical Record Review Results Summary Sheet below, and stop.
If the event is present on both the medical and encounter records, proceed to validation of the specified data fields.
Required Review: (Check One)
[

] Office visit: Includes all services, except dental and mental health/substance abuse

[

] Office visit: Includes mental health/substance abuse services only

[

] Office visit: Includes dental services only

[

] IP admission: Includes all IP services, except mental health/substance abuse services

[

] IP admission: Includes mental health/substance abuse services only

[

] Other types of encounters utilized by the state (e.g., lab/x-ray; physical therapy)

[

] Specify other service type: _______________________

PROTOCOL FIVE | 219

Data Field Validation Table
Diagnosis codes and descriptors
Encounter line #

Encounter Dx
code

Dx description

Medical record
Dx code

Match

No match

Procedure
description

Medical record
procedure code

Match

No match

Revenue
description

Medical record
revenue code

Match

No match

1
…
N
Procedure codes and descriptors
Encounter line #

Encounter
procedure code

1
…
N
Revenue codes and descriptors
Encounter line #

Encounter
revenue code

1
…
N
Note: The EQRO should edit this table to include all data elements under review.

220 | PROTOCOL FIVE

Worksheet 5.6. Medical Record Review Results Summary Sheet
Research question:
_____________________________________________________________________________________________
_____________________________________________________________________________________________
_____________________________________________________________________________________________
Sample size: ___________
Sampling methodology: __________________________________________________________________________
Please summarize how the MCP addresses medical record review auditing or accuracy checks:
_____________________________________________________________________________________________
_____________________________________________________________________________________________
Record of substitutions (list substitutions and reasons):
Original Record

Replacement Record

Replacement Reason

1.
2.
3.
4.
Results:
Record
numbers
reviewed

Event noted on encounter
record

Event recorded
in medical
record

Match? (Yes / No)

Notes / Comments

1
2
3
4
5
…
N
Error rate (total records with errors/total records in sample): ____________
Reviewer summary of findings (including reasons for errors):

PROTOCOL FIVE | 221

Worksheet 5.7. Suggested Format for Reporting Encounter Data Validation Information in
the EQR Technical Report
Instructions. Use Worksheet 5.7 as a framework to report findings from the encounter data validation activities in
Protocol 5 by MCP.
For each MCP, please complete the following information:
MCP name
MCP contact name and title
Mailing address
Phone/fax numbers
Email address
EQRO interview date
Type of delivery system (check all that apply)

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM □ LTSS
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

Note: IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care Organization;
PIHP = Prepaid Inpatient Health Plan; PCCM = Primary Case Management.

Encounter Type

Records Received
and Reviewed

Total Elements
Possible

Total Matched
Elements

Percentage of
Matched
Elements

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

Inpatient
Outpatient
Office visit
Total

Inpatient Encounter
Type
Match
No Match
Total Elements
Match Percent
Outpatient
Encounter Type
Match
No Match
Total Elements
Match Percent

222 | PROTOCOL FIVE

Office Visit
Encounter Type

Diagnosis Codes

Procedure Codes

Revenue Codes

Total

Match
No Match
Total Elements
Match Percent

No Match for Diagnosis Code Element

Encounter Type

Total Elements

Lack of Medical Record
Documentation

Incorrect Principal
Diagnosis (Inpatient) or
Incorrect Diagnosis
Codes

Inpatient
Outpatient
Office Visit
Total

No Match for Procedure Code Element

Encounter Type

Total Elements

Lack of Medical Record
Documentation

Incorrect Principal
Diagnosis (Inpatient) or
Incorrect Diagnosis
Codes

Inpatient
Outpatient
Office Visit
Total

No Match for Revenue Code Element

Encounter Type

Total Elements

Lack of Medical Record
Documentation

Incorrect Principal
Diagnosis (Inpatient) or
Incorrect Diagnosis
Codes

Inpatient
Outpatient
Total

END OF WORKSHEETS FOR PROTOCOL 5

PROTOCOL FIVE | 223

This page left blank for double-sided copying.

224 | PROTOCOL FIVE

PROTOCOL 6. ADMINISTRATION OR
VALIDATION OF QUALITY OF CARE SURVEYS
AN OPTIONAL EQR-RELATED ACTIVITY
SECTION I. ADMINISTERING A SURVEY
ACTIVITY 1: IDENTIFY THE SURVEY PURPOSE, OBJECTIVES, AND
AUDIENCE

ACTIVITY 3: SELECT THE SURVEY INSTRUMENT
ACTIVITY 4: DEVELOP THE SAMPLING PLAN
ACTIVITY 5: DEVELOP A STRATEGY TO MAXIMIZE RESPONSE
ACTIVITY 6: DEVELOP A QUALITY ASSURANCE PLAN
ACTIVITY 7: IMPLEMENT THE SURVEY ACCORDING TO THE WORK PLAN
ACTIVITY 8: PREPARE AND ANALYZE SURVEY DATA AND PRESENT
RESULTS IN A FINAL REPORT
SECTION II. VALIDATING A SURVEY
ACTIVITY 1: REVIEW THE SURVEY PURPOSE, OBJECTIVES, AND
AUDIENCE
ACTIVITY 2: REVIEW THE WORK PLAN
ACTIVITY 3: REVIEW THE RELIABILITY AND VALIDITY OF THE
SURVEY INSTRUMENT
ACTIVITY 4: REVIEW THE SAMPLING PLAN
ACTIVITY 5: REVIEW THE ADEQUACY OF THE RESPONSE RATE
ACTIVITY 6: REVIEW THE QUALITY ASSURANCE PLAN
ACTIVITY 7: REVIEW THE SURVEY IMPLEMENTATION
ACTIVITY 8: REVIEW THE SURVEY DATA ANALYSIS AND FINAL REPORT

BACKGROUND
Surveys are an important resource for assessing the experience of
managed care enrollees and providers. Information derived from surveys
can help states and managed care plans (MCPs) create a personcentered health care environment for those enrolled in Medicaid and the
Children’s Health Insurance Program (CHIP). Enrollee surveys can be
used to assess experience with their health plan and its providers, and
the quality of care they receive. Provider surveys can be used to assess
the characteristics of providers and practices that serve Medicaid/CHIP
enrollees, their accessibility and availability, and their experience with the
Medicaid/CHIP program.

PROTOCOL SIX

ACTIVITY 2: DEVELOP A WORK PLAN

225

This protocol provides guidance for administering and validating consumer or provider surveys.
These surveys may be administered by states or MCPs (or their vendors) and validated by an
External Quality Review Organization (EQRO) or administered by the EQRO on behalf of a
state or MCP. Because this protocol may be used for a variety of purposes, it does not specify
one survey instrument, sampling method, or analytical approach.
An overarching goal of this protocol is to provide guidance about designing and conducting
surveys that produce valid and reliable results. In this context, validity refers to surveys that
measure what they were intended to measure. Reliability refers to the internal consistency of a
survey and the reproducibility of survey results when administered under different conditions
(e.g., by different people or at different times). Please refer to the Technical Appendix at the end
of this protocol for further discussion about potential sources of survey error that can affect the
overall quality of a survey.

GETTING STARTED ON PROTOCOL 6
This protocol includes eight activities related to administering and validating a survey (Figure
6.1). When an EQRO validates a survey, the activities in 6.1 focus on ensuring the survey was
administered correctly. Although the focus of the EQR-related activity will differ depending on
whether the EQRO’s role is to administer or validate a survey, these eight activities are common
to both roles.

226 | PROTOCOL SIX

Figure 6.1. Protocol 6 Activities

Note: These activities pertain to survey implementation. Survey validation activities involve reviewing the adequacy of survey
implementation.

PROTOCOL SIX | 227

Two supplemental resources are available to help EQROs administer and validate a survey:
•

•

Worksheets for Protocol 6. Survey Administration and Validation Tools, which can be used
to guide the EQRO’s activities as follows:
○

For survey administration: Use the worksheets to track and document steps performed
in designing and implementing the survey. In the “Comments” column, document
decisions or findings

○

For survey validation: Use the worksheets to track and document steps performed in
validating the survey. In the “Comments” column, document the outcome of validation
activities, including sources reviewed. The worksheets can also be used as an outline
for the final report to the state. Expand the tool to include other activities or findings as
needed

Appendix B. Sampling Approaches for EQR Data Collection Activities, which provides an
overview of sampling methods

Section I of this protocol describes the activities associated with administering a survey. Section
II describes the activities associated with validating a survey.

SECTION I. ADMINISTERING A SURVEY
ACTIVITY I.1: IDENTIFY THE SURVEY PURPOSE, OBJECTIVES, AND
AUDIENCE
WORKSHEET 6.1

The first step in developing a survey is to identify the
survey purpose, objectives, and audience (Worksheet
6.1). The EQRO should develop a clear understanding of
how a state will use the survey results, including what
the state wants to learn from the survey and what it
plans to do with the results (See box, Examples of
Survey Uses).

Resources for Activity 1
Worksheet 6.1. Survey Purpose,
Objectives, and Audience
• Provides a set of questions to assess
the clarity of the survey purpose,
objectives, and audience

The state should also specify the audience for the
findings, since the survey content, analysis plan, and report format will vary based on the
audience. Such audiences and uses could include the following:

228 | PROTOCOL SIX

•

Enrollees and their families. Increasingly,
consumers rely on survey information to inform their
choice of health care options. To support this use,
the survey design must allow for comparisons
among MCPs, potentially controlling for or stratifying
enrollee characteristics
MCPs and providers. To promote value-based
purchasing in Medicaid and CHIP, information on
the quality of care provided by MCPs and providers
can be used to identify higher- and lower-performing
plans and practices and support quality
improvement initiatives

Examples of Survey Uses
• Monitor and evaluate access,
timeliness, and quality of care
provided to Medicaid and CHIP
enrollees
• Inform value-based purchasing and
quality improvement initiatives
• Provide information to help Medicaid
and CHIP enrollees make informed
choices among MCPs

State policymakers. With increasing recognition of the link between better care, better
health, and more affordable care, information on enrollee experiences in managed care,
barriers to care, and the role of social determinants can be used to develop initiatives to
reduce disparities and improve outcomes

Next, the state should specify survey domains that align with the intended use of the survey
results. For example, if the survey results will be used to help enrollees choose a health plan,
specific measurement domains might include experience with the primary care provider, access
to specialty care, and treatment planning, among others.
Finally, the state should specify the unit of analysis, including populations or subpopulations of
interest. Depending on the purpose of the survey, the unit of analysis could be the entire
managed care population in the state or it could be targeted to subpopulations, for example,
individual MCPs, provider groups, children with chronic conditions, new Medicaid enrollees, or
individuals recently disenrolled from an MCP. This information is used to develop the sampling
approach, instrument design, and analysis plan.

ACTIVITY I.2: DEVELOP A WORK PLAN
WORKSHEET 6.2

After determining the intended use of the survey in
Resources for Activity 2
collaboration with the state, the EQRO should prepare a
work plan that will govern the implementation of the
Worksheet 6.2. Work Plan
survey (including the project management plan,
• Provides a set of questions to assess
schedule, and reporting requirements). Key issues to
the work plan
address in the work plan are summarized in Worksheet
6.2. Refer to Activity I.6 for examples of typical weekly
data collection schedules. The EQRO should obtain state approval of the work plan before
implementing the survey and then administer the survey in accordance with the approved work
plan.

PROTOCOL SIX | 229

ACTIVITY I.3: SELECT THE SURVEY INSTRUMENT
WORKSHEET 6.3

The state’s choice of a survey instrument should be
consistent with the purpose of the data collection, unit of
analysis, and goal of collecting valid and reliable data
(Worksheet 6.3). This protocol describes three options
for selecting the survey instrument:

Resources for Activity 3
Worksheet 6.3. Survey Instrument
• Provides a set of questions to assess
the selection of the survey instrument

•

Option 1. Use an existing validated survey
instrument

•

Option 2. Adapt an existing survey instrument with additional state-specific questions

•

Option 3. Develop a new survey instrument

The state may choose among the three options independently or in consultation with the EQRO.
However, there often are trade-offs in selecting an instrument. Use of an existing instrument
may provide the greatest assurance of validity and reliability but omit certain key domains of
interest. In contrast, development of a new survey instrument may provide the closest alignment
with the intended use of the survey results but validity and reliability may be untested. Thus,
another option is to adapt an existing survey instrument by adding state-specific questions to
address gaps in survey content. These three options are summarized in more detail below.
Option 1. Use an Existing Validated Survey Instrument
Use of an existing well-validated instrument offers several benefits such as:
•

Readily accessible

•

Cost efficient

•

Minimal development and testing hours

•

Often translated into Spanish or other languages

•

Available benchmark data that can be used for context and comparisons

•

Potential for rapid launch of data collection to investigate time-sensitive issues

The state or EQRO can select from a variety of existing survey instruments. Table 6.1 provides
examples of instruments that have been used to gather (1) beneficiary feedback about
experiences with health care, and (2) provider feedback on organizational issues.

230 | PROTOCOL SIX

Table 6.1. Examples of existing validated survey instruments
Example

Description

Beneficiary surveys
CAHPS®

• Developed by AHRQ in collaboration with the CAHPS® Consortium, the CAHPS®
survey instruments and reporting formats have undergone rigorous testing for reliability
and validity
• States frequently use the CAHPS® surveys to assess enrollees’ experiences with
managed care; versions include a Health Plan Survey; Clinician & Group Survey;
Hospital Survey; and Cancer Care Survey. An overview is available at
https://www.ahrq.gov/cahps/index.html
• Includes surveys for Adult and Child Medicaid enrollees available at
https://www.ahrq.gov/cahps/surveys-guidance/hp/index.html
• Allows for the addition of supplemental questions; see
https://www.ahrq.gov/cahps/surveys-guidance/item-sets/search.html
• National and regional benchmarks are available; information about the CAHPS
database is available at https://www.ahrq.gov/cahps/cahps-database/index.html
• Note that there are two versions of CAHPS (AHRQ and NCQA); more information
about the differences between the two versions is available at
https://www.ahrq.gov/cahps/surveys-guidance/hp/about/NCQAs-CAHPS-HPSurvey.html

The Mental Health
Consumer Perception
Survey

• Designed in the 1990s to report consumers’ experience with the quality of mental
health programs and service delivery
• Available for both adults and children
• Benchmarks are available for comparison
• The Mental Health Quality Statistics Improvement Program (MHSIP) Quality Report
Toolkit is available at http://www.hsri.org/publication/the-mhsip-quality-report-toolkit

Patient satisfaction
questionnaires

• Professional associations such as MGMA, AAFP, and state medical societies may
provide existing patient satisfaction questionnaires to assess beneficiary experience

Provider and practice surveys
Patient Centered
Medical Home
Assessment
(PCMH-A)

• Standardized practice-level survey instrument
• Designed to help practices monitor their progress as they transition to a medical home
care model and identify areas for improvement
• More information is available at
http://www.safetynetmedicalhome.org/sites/default/files/PCMH-A.pdf
• A modified PCMH-A survey was developed for the Comprehensive Primary Care
Initiative (PCPI) evaluation; for more information see
https://innovation.cms.gov/Files/reports/cpci-evalrpt2.pdf

Staff Experience
Survey

• Developed by the University of Chicago to assess staff experience across multiple
domains: access to care and communication with patients, tracking data, electronic
medical record, care management, quality improvement, work satisfaction, work
environment, work activities, and demographics
• Survey instrument is available at
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3752653/

Note: AAFP = American Academy of Family Physicians; AHRQ = Agency for Healthcare Research and Quality; CAHPS® =
Consumer Assessment of Healthcare Providers and Systems; MGMA = Medical Group Management Association; NCQA = National
Committee for Quality Assurance.

PROTOCOL SIX | 231

Even when using an existing instrument, the
EQRO should review the instrument’s reliability
and validity based on published or unpublished
documentation (See box, Validity and Reliability).
For example, existing validated survey instruments
may not have been validated in a Medicaid or
CHIP population or not tested in languages other
than English. Selecting instruments not validated in
the target population may not yield valid or reliable
results for that population. When using an existing
survey instrument, the EQRO should document
findings related to reliability and validity testing of
the survey instrument, preferably in a comparable
population.

Validity and Reliability
Validity refers to the degree to which the survey
is measuring what was intended to be
measured and is made up of two components:
• Face validity refers to the degree to which
the survey is measuring what was intended
to be measured
• Content validity refers to whether the survey
questions accurately represent the concept
or subject matter being measured
Reliability refers to:
• Internal consistency of a survey
• Reproducibility of survey results when
administered under different conditions (such
as by different people or at different times)

Note that if a state or EQRO chooses to administer
a CAHPS® survey, the CAHPS® Health Plan
Fielding Guide contains step-by-step instructions
for drawing a sample, administering the survey, and analyzing the data. The Guide is available
at https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/hp/fielding-thesurvey-hp50-2013.pdf.
Option 2. Adapt an Existing Survey
Another option is to adapt an existing survey by adding or deleting items, modifying questions,
or using only certain groups of questions relevant to the state’s survey objectives. Modifying an
existing questionnaire provides the state with the flexibility to add or change the survey content
while providing many of the advantages of using a pre-existing questionnaire. However, adding,
deleting, or modifying questions may undermine the validity and reliability of the questions, as
well as the survey overall. Validated questionnaires are tested “as a whole,” and modifications
can change the focus and purpose of the questionnaire.
Some surveys, such as the CAHPS® Medicaid Health Plan Survey, provide optional
supplemental questions the state can consider using to customize the questionnaire. This has
the advantage of providing a validated instrument that allows comparisons, while
accommodating special questions of particular interest to the state or MCPs.
When the EQRO adapts an existing questionnaire, it should consult with an expert in survey
design about incorporating the modification and conducting appropriate tests for reliability and
validity. Any new translations should also be tested.
Option 3. Develop a New Survey Instrument
The state or EQRO may also decide to develop a new survey instrument when the survey
purpose requires answers to questions not measured by existing instruments. A well-designed
instrument can capture information of interest and relevance to the questions under study.

232 | PROTOCOL SIX

The state or EQRO should follow best practices
in designing the instrument (See box, Best
Practices in Questionnaire Design). If possible,
the state or EQRO should involve a survey
design expert to address issues associated with
respondent burden, comprehension, and
readability.
In addition, the state or EQRO should work with
a survey design expert to assess face and
content validity and conduct a pretest of the
instrument for reliability (See box, Validity and
Reliability, above). Although assessment of the
validity and reliability of new surveys can be
costly and time consuming, such testing is key
to identifying methodological flaws that could
make the results suspect.

Best Practices in Questionnaire Design
• Questions are worded clearly and briefly, and in
an unbiased manner so respondents can readily
understand key terms and concepts
• Questions request information that respondents
can reasonably be expected to report
• Question response categories are appropriate,
mutually exclusive, and reasonably exhaustive
given the intent of the questions
• Questions are accompanied by clear, concise
instructions and probes so that respondents will
know exactly what is expected of them
• All questions can be easily understood by
someone with a sixth-grade reading level

Face and content validity can be assessed by conducting cognitive interviews or convening one
or more focus groups that include targeted survey respondents or individuals with subject matter
expertise. A factor analysis could also be conducted to verify that the individual items that
comprise a scale are measuring the domain of interest. Reliability can be assessed using the
test-retest method in which the survey is administered to the same group at two different times.
A correlation coefficient is calculated and indicates the reproducibility of results. Correlation
coefficients with r-values at or above 0.70 indicate good reliability. However, even with high
reliability, a new survey instrument will have limited benchmarks for comparison of results.

ACTIVITY I.4: DEVELOP THE SAMPLING PLAN
WORKSHEET 6.4

The EQRO should develop a sampling plan that
represents all eligible enrollees within the MCP
(Worksheet 6.4). Refer to Appendix B. Sampling
Approaches for EQR Data Collection Activities for an
overview of sampling approaches that can be used for
drawing a survey sample. In general, the sampling plan
should incorporate information from the five steps
described below.
Step 1: Define the Study Population

Resources for Activity 4
Worksheet 6.4. Sampling Plan
• Provides a set of questions to assess
the sampling plan
Appendix B. Sampling Approaches for
EQR Data Collection Activities
• Provides an overview of sampling
approaches and guidance for
determining sample sizes for EQR
data collection activities

The EQRO must first define the population to be studied
(for example, all Medicaid or CHIP beneficiaries enrolled
in an MCP or all children with chronic conditions) and
then determine which data source(s) to use to construct a list of all units in the study population
(this list is referred to as the sampling frame). The sampling frame will be used to draw the
sample for data collection. The sampling frame should include all the information necessary to
determine whether units in the population are eligible for the study (e.g., dates of Medicaid

PROTOCOL SIX | 233

coverage and MCP enrollment) and any information that would be used for stratification by
subgroup (e.g., age, gender, zip code of residence).
Step 2: Determine the Type of Sampling to be Used

There are two basic types of sampling methods.
•

Probability (or random) sampling methods leave selection of population units to chance
and not to convenience or preference on the part of the individuals conducting the study or
otherwise participating in the study. Probability sampling removes systematic bias in the
selected sample due to observed and unobserved differences in the sampling units

•

Non-probability sampling methods are used when subjects are scarce or hard to sample
(no sampling frame) and/or the study relies on volunteers. The sample is based on the
choice of those administering the survey rather than chance; therefore, some bias can be
expected

Probability sampling is preferable to non-probability sampling when feasible because it removes
systematic bias from the sample. For more information on commonly used types of probability
and non-probability sampling methods, see Appendix B.
Step 3: Determine the Number of Units to Sample

The number of units selected in the sample depends on several factors, including the level of
precision required to achieve statistically valid results, the expected number of respondents (i.e.,
the response rate), and other constraints on the financial and personnel resources available to
administer the survey. Samples with a larger number of units will provide a higher level of
precision, but may be more expensive to collect data from and present more of a burden on
financial and personnel resources.
For the CAHPS® Medicaid Health Plan Survey, research has determined that that 300
completed surveys per plan or product will provide statistically valid results. Thus, if the EQRO
estimates that 50 percent of the sampled individuals will complete the CAHPS® Medicaid
Health Plan Survey, then approximately 600 surveys must be fielded to reach the 300
completed surveys. For other surveys, the EQRO should consider contacting a sampling
statistician to conduct a statistical power analysis to help determine the optimal number of units
to sample to meet precision targets while accounting for financial and personnel burden.
Given the interdependence between sample size, response rate, and precision, a goal is to
achieve the highest response rate possible. See Activity I.5 for strategies to maximize survey
response.
Step 4: Select the Sample

In this step, the EQRO determines how the sample will be selected. For probability sampling
methods, the sample can be drawn using statistical software packages. For non-probability
samples, the sample is selected by the EQRO based on convenience or perceived
representativeness of the study population. The sampling plan should clearly explain the
sampling methods, and describe the procedures used to minimize bias. For more information on
selecting the sample, see Appendix B.

234 | PROTOCOL SIX

ACTIVITY I.5: DEVELOP A STRATEGY TO MAXIMIZE RESPONSE
WORKSHEET 6.5

The EQRO should develop a strategy for maximizing
survey response that includes a plan for both locating
and contacting the sample members.
Step 1: Maximize Completeness of Sample
Information Before Survey Launch

Resources for Activity 5
Worksheet 6.5. Strategy to Maximize
Response
• Provides a set of questions to assess
the strategy for locating sample
members and specific data needed to
administer the survey

Before the survey is implemented, the EQRO should
identify the specific data it needs to locate sample
members and develop a strategy for ensuring the
locating information is complete. The following
information is frequently used to locate sample members in Medicaid surveys:
•

First and last name

•

Address

•

Home and cell phone numbers

•

E-mail address

•

Date of birth

•

Primary language

•

Preferred language

•

Name of MCP

•

Length of enrollment

These data elements should be used for the purpose of contacting sample members and should
be kept separate from the survey data to protect the confidentiality of the sample members’
survey responses and protected health information (PHI). The survey data provided by the
sample member should be identified by a unique, numeric identification number, not by name or
other identifying characteristic.
The EQRO should collect complete contact information and consider that some information may
be verified through the state’s eligibility files or the MCPs’ enrollee files. The EQRO may also
need to establish a data use agreement with the state or MCP for the protection and handling of
PHI. The EQRO should also expect missing data in the state data files and document its plans
to locate and contact respondents, including sending names in the sample file to a telephone
number look-up vendor or using a change-of-address database vendor.

PROTOCOL SIX | 235

Step 2: Design a Data Collection Strategy that Maximizes Response

The EQRO should design a data collection strategy that
maximizes response and fits within the available budget
and schedule (See box, Tips for Response Rate). The
data collection strategies described below represent best
practices in the field of survey research and are
frequently used to maximize survey response. 67 The
EQRO should design a data collection plan that uses
some or all of the strategies described below:

Tips for Response Rates
While no response rate can be
guaranteed, the developers of CAHPS®
suggest that employing contact
strategies and nonresponse follow-up
plans should yield a response rate
between 40 and 50 percent.

Advance letter. Including an introductory letter before starting data collection lends
legitimacy to the survey. A good letter emphasizes survey sponsorship (e.g., on state
government letterhead signed by the agency director), describes the purpose of the survey,
includes a statement about sample member confidentiality, provides information on how the
sample member was selected for the survey, and describes benefits to the sample member
as a result of participation (and emphasizes there is no penalty for not responding). In
addition, the letter should be personalized and addressed to the respondent by name

Multiple and varied call attempts. Best practices to increase survey response include the
use of varied contact attempts. This can include multiple contact attempts at different times
of the day, mailing a reminder postcard or second survey if conducting a self-administered
paper survey, making a follow-up phone call to non-respondents to a mail survey, or
conducting repeat calls in a telephone survey. The EQRO should track and follow up on the
number of respondents that could not be contacted or failed to respond

Multi-mode surveys. Combining two or more modes of data collection (such as mail and
phone) in a single survey effort can lead to higher response rates than single-mode
surveys. This is because multi-mode surveys may:

○

Lower costs by beginning data collection in a cost-effective mode

○

Allow the data collection to continue for longer periods

○

Increase the timeliness of response (for example, results from a web-based survey can
be received faster than results by mail)

○

Limit coverage error, for example, by offering the survey by mail or web if the
population of interest may not have consistent telephone service. In addition, as more
households have access to the internet, using email and text messaging (with
respondent permission) has become an increasingly common method to contact
respondents (See box, Integrating Web-Based Outreach in Data Collection, next page)

Multiple languages. The state and MCPs should have information about each
beneficiary’s primary and/or preferred language. If this information is not readily available,
the EQRO might include a sentence in the advance letter translated into the most common
languages in the area, inviting the individual to call for more information or to request a
specific translation

For more information about best practices see: (1) Dillman, Don A., Jolene D. Smyth, Leah Melani Christian. Internet, Mail
and Mixed-Mode Surveys: The Tailored Design Method (4th ed.), 2014. (2) Groves, R.M. Non response rates and
nonresponse bias in household surveys. Public Opinion Quarterly, vol. 70, no. 5, Special Issue 2006, pp. 646–675.

236 | PROTOCOL SIX

Integrating Web-Based Outreach in Data Collection
• Send the advance letter by email in place of, or in addition to, first class mail
• Send the reminder postcard by email in place of, or in addition to first class mail
• Include a hyperlink to an online version of the survey in the advance letter or advance email
• Begin data collection with an online survey and follow-up with non-respondents by mail and/or telephone
• When deciding which mode(s) to use, always consider the population's access to and preferences for each mode

The strategies should be tailored to the survey population. In particular, the EQRO should
customize strategies for provider surveys. While the design and contact strategies listed above
also are effective when surveying health care professionals (such as physicians, nurses, and
practice staff), health care professionals historically are a difficult population to reach. A metaanalysis 68 of 154 surveys of health care professionals found statistically significant
improvements in response rates when using mail-based data collection (compared to the webbased mode) and when monetary incentives were offered (compared to those that did not offer
a monetary incentive or offered a non-monetary incentive). In addition, surveys with one or two
follow attempts yielded higher response rates than studies with three or more follow-ups.
Step 3: Specify the Method Used to Calculate the Response Rate

The sampling plan should specify the method that will be used to calculate the response rate.
The EQRO should use a standard methodology to calculate the response rate. The American
Association of Public Opinion Research (AAPOR) provides a list of standard definitions and
response rate calculators on its website at http://www.aapor.org/Standards-Ethics/StandardDefinitions-(1).aspx.
The sampling plan should also note target response rates for similar surveys, which can be
used as a benchmark to assess the adequacy of the response rate after the survey is
implemented, such other surveys conducted by the state or by other states, or other types of
surveys implementing the same methodology.
Step 4: Include a Plan for a Non-Response Analysis

Finally, after the survey is complete, a non-response analysis should be conducted, as discussed
in Activity I.8. The sampling plan should describe the approach that will be taken to a non-response
analysis to assess whether there are differences between respondents and non-respondents.

ACTIVITY I.6: DEVELOP A QUALITY ASSURANCE PLAN
WORKSHEET 6.6

The EQRO should develop a quality assurance plan
that contains quality checks for all phases of the data
collection effort (Worksheet 6.6). The quality
assurance plan should describe the checks to be
performed and the processes used to implement the

Resources for Activity 6
Worksheet 6.6. Quality Assurance Plan
• Provides a set of quality check questions
to assess the quality assurance plan

68 Young Ik Cho, Timothy P. Johnson, and Jonathan B. VanGeest. Enhancing Surveys of Health Care Professionals: A MetaAnalysis of Techniques to Improve Response. Evaluation & the Health Professions, vol. 36, no. 3, pp. 382-407, August 2013.

PROTOCOL SIX | 237

checks (See box, Tips for Quality Assurance Checks). The quality checks should cover the
sampling and locating processes, be customized by data collection mode, and specify data
quality controls.
Tips for Quality Assurance Checks
The quality assurance plan should describe each
quality check and clearly identify:
• What checks are being performed
• How the checks are performed
• Who performs the checks
• Frequency of the checks
• Percentage of survey records that are to be checked
• Corrective actions required if an issue is identified
• How the issue was resolved

ACTIVITY I.7: IMPLEMENT THE SURVEY ACCORDING TO THE WORK
PLAN
WORKSHEET 6.7

The EQRO should implement the survey according to
the weekly data collection schedule laid out in the work
plan (see Activity I.2 and Worksheet 6.7). Although there
is no set time frame for data collection, on average, data
collection activities range from 10 to 14 weeks. Three
sample data collection schedules by week are included
in Table 6.2. Any deviations from the work plan should
be documented and the reasons for those deviations
should be explained.

238 | PROTOCOL SIX

Resources for Activity 7
Worksheet 6.7. Survey Implementation
According to the Work Plan
• Provides a set of questions to assess
survey implementation

Table 6.2. Sample data collection schedules by week
Data
collection
week

Mail-only protocol

Telephone-only protocol

Mixed-mode example:
Mail with telephone follow-up

Mail initial survey with cover
letter to sample members

Mail advance letters and begin
telephone contact to sample
members

Mail initial survey with cover
letter to sample members

Mail optional postcard and
receipt returned surveys

Mail optional postcard and
continue telephone follow-up

Mail optional postcard and
receipt returned surveys

Receipt returned surveys

Continue telephone follow-up
(weeks 3–10)

Receipt returned surveys

Mail second survey with cover
letter to non-respondents

Receipt returned surveys
(weeks 5–10)

Receipt returned surveys

Telephone follow up to nonrespondents and receipt
returned surveys (weeks 6–12)

End data collection

ACTIVITY I.8: PREPARE AND ANALYZE SURVEY DATA AND PRESENT
RESULTS IN A FINAL REPORT
WORKSHEET 6.8

Once the surveys have been completed and returned,
the EQRO must prepare the data for analysis. This may
include post-processing procedures (e.g., cleaning and
editing, creating weights, and conducting a nonresponse
analysis). Then the EQRO proceeds with the data
analysis in accordance with the work plan and prepares
the final report (Worksheet 6.8).

Resources for Activity 8
Worksheet 6.8. Survey Data Analysis
and Final Report
• Provides a set of questions to assess
the data analysis and final report

Step 1: Implement Post-Processing Procedures

Consistent with the quality assurance plan (Activity I.6), the EQRO should implement
procedures to handle responses that fail edit checks, address missing data, and remove data
from surveys determined to be unusable. The EQRO should specify the criteria used to remove
surveys or data from the final analytic file (including the threshold used to determine a
completed case). The EQRO should document the reasons for all exclusions or adjustments of
data used for the analysis.

PROTOCOL SIX | 239

Step 2: Calculate the Sampling Weights

When a sample is selected in such a way that there are different probabilities of selection for
different units, sampling weights must be constructed and used for any analyses conducted with
data collected from the sample. The weights take into account the sample design and
nonresponse of sampled units. The weighted results, therefore, are representative of the
population not just the units that responded to the survey.
The sampling weight is equal to one over the probability of selection for a unit. For example, if
the probability of selection is 0.25 for a unit, the sampling weight is 1/0.25 = 4. In probability
sampling, the sampling weights are used to make inferences to the study population. The
sampling weights would also need to be adjusted to account for nonresponse if there is
considerable nonresponse during data collection. The EQRO should consult a sampling
statistician to help calculate the sampling weights.
Step 3: Conduct a Non-Response Analysis

The response rate is only one indicator of survey quality. Another indicator is the extent to which
non-respondents may differ from respondents on the key variables in the survey sample.
Because significant differences may bias the survey estimates, it is important to conduct a
nonresponse analysis to assess the representativeness of the survey respondents.
Before beginning the data analysis, the EQRO should compare the characteristics of
respondents and non-respondents using means and frequency distributions. The analysis
should rely on information available in the sample frame (such as information found in state
Medicaid eligibility files). Tests of statistical significance (e.g., t-tests and chi-square tests)
should be performed to determine whether the differences are statistically significant. If there
are substantial differences between respondents and non-respondents, the EQRO should
consult a statistician to assess what types of adjustments might be necessary to account for
potential bias in the survey responses.
Step 4: Analyze Survey Data

Following the analysis plan laid out in the work plan and approved by the state, the EQRO
should generate means or frequency distributions for each survey question and calculate
statistics. The analysis should include a description of the population characteristics,
performance on the outcome measures included in the survey (such as access, timeliness, and
quality of care or experience of care).
In addition, the EQRO should examine differences in survey results among MCPs, between
MCPs and the FFS or PCCM population (if applicable), or between MCPs in the state and
nationally or regionally (if benchmarks are available). The EQRO could also analyze and report
on variations among subpopulations within each MCP. For example, the state may be interested
in whether responses differ significantly across geographic locations, racial/ethnic groups,
socioeconomic groups, or other identifiable subgroups. For recurring surveys with trendable
results, the EQRO could examine changes over time on key metrics.
Results should be weighted, account for the complex sample design in computing variances (if
applicable), and take into consideration the adequacy of sample sizes to support the analyses.
Some surveys include open-ended, qualitative responses related to experience or satisfaction.
In such cases, the open-ended responses should be reviewed, coded into categories if feasible,
240 | PROTOCOL SIX

and synthesized for analysis. Such information can enrich the quantitative data analysis and
provide a “voice” to illustrate the numerical findings.
Step 5: Prepare and Submit a Final Report

The EQRO should prepare and submit reports in the agreed format, which may include:
•

Survey purpose and objectives

•

Survey implementation procedures, including challenges encountered, lessons learned,
and recommendations for improving future efforts

•

Overview of analytic findings, including subgroup analyses and tests of statistical
significance

•

Methodologically appropriate, comparative information about MCP performance

•

A detailed assessment of each MCP’s strengths and weaknesses with respect to access,
quality, and/or timeliness of health care furnished to enrollees

•

Conclusions drawn from the data

Results from the survey should always be presented for groups and not for individual
respondents. Statistical graphs should accompany narrative text to aid comparison and
interpretation. For example, bar graphs and comparison charts, such as those recommended by
CAHPS®, convey important information about the performance of each MCP and indicate
meaningful differences among MCPs.
The EQRO should submit a draft report and provide the state with an opportunity to review and
comment on the draft report. The EQRO should then revise the draft and submit a final report
that incorporates state comments. Other deliverables may include a raw data file and analysis
files, as well as public reports, presentations, or web sites developed for public reporting.

SECTION II. VALIDATING A SURVEY
Protocol 6 also contains guidance for EQROs charged with validating a survey conducted by a
state, MCP, or a vendor hired by the state or MCP. 69 The activities described in this section
focus on reviewing the survey design and implementation for validity, reliability, and
methodological rigor. They do not include collecting additional survey data from survey
respondents to verify their responses or test for survey validity and reliability.
The EQRO should use the Worksheets for Protocol 6 or a similar tool to guide the validation
process. The EQRO should identify the documentation it used to review the survey procedures
and note its findings for each activity. In addition, the EQRO should note the absence of
documentation for a particular activity as it may be relevant to the survey validation.
Upon completion of the validation activities, the EQRO should synthesize all of the validation
findings from Activities II.1 through II.8 based on the findings documented in the Worksheets for
Protocol 6. The EQRO should submit a final validation report that assesses the overall quality of
the survey, and in particular, the extent to which the survey achieved its purpose and objectives.
Many states and MCPs contract with survey vendors certified by the National Committee for Quality Assurance (NCQA) to
conduct CAHPS® 5.0H surveys following a standardized and validated protocol. A list of approved vendors is available at
http://www.ncqa.org/hedis-quality-measurement/data-reporting-services/cahps-5-0-survey.

PROTOCOL SIX | 241

Key elements of this assessment are whether the survey findings can be generalized to the
population from which the sample was drawn and whether the data quality and completeness
can support the survey’s intended uses.
Although survey validation is an optional EQR-related activity, CMS recommends that surveys be
validated when states intend to use survey results for such decisions as consumer health plan
selection, health plan or provider payment, or performance incentives (e.g., auto-assignment).

ACTIVITY II.1: REVIEW THE SURVEY PURPOSE, OBJECTIVES, AND
AUDIENCE
WORKSHEET 6.1

To understand and evaluate the adequacy of the survey
to meet its intended uses, the EQRO should seek
information from written sources or through interviews
about the survey’s purpose, objectives, and audience.
See Activity I.1 for more information about defining the
survey purpose, objectives, and audience.

Resources for Activity 1
Worksheet 6.1. Survey Purpose,
Objectives, and Audience
• Provides a set of questions to assess
the clarity of the survey purpose,
objectives, and audience

ACTIVITY II.2: REVIEW THE WORK PLAN
WORKSHEET 6.2

To understand the survey implementation plan, the
Resources for Activity 2
EQRO should review the work plan, including the project
management plan, schedule, reporting requirements,
Worksheet 6.2. Work Plan
data preparation plan, data analysis plan, and security
• Provides a set of questions to assess
protocols and procedures. The work plan provides a
the work plan
foundation for understanding the rigor of the overall
survey approach; deviations from the work plan may
signal concerns related to the effectiveness of survey implementation. See Activity I.2 for more
information about developing a work plan.

ACTIVITY II.3: REVIEW THE RELIABILITY AND VALIDITY OF THE
SURVEY INSTRUMENT
WORKSHEET 6.3

As discussed in Activity I.3, there are three options for
selecting a survey instrument:
1

Use an existing validated survey instrument

Adapt an existing survey instrument with additional
state-specific questions

Develop a new survey instrument

242 | PROTOCOL SIX

Resources for Activity 3
Worksheet 6.3. Survey Instrument
• Provides a set of questions to assess
the selection of the survey instrument

Each of these approaches involves trade-offs. For example, use of an existing validated survey
instrument increases assurances about the instrument’s validity and reliability, but the
instrument may have gaps in survey content for the specific survey purpose. Development of a
new survey instrument may result in more targeted content for the specific survey purpose, but
require more effort to ensure validity and reliability of the instrument. As part of this validation
activity, the EQRO is charged with assessing the extent to which there is sufficient
documentation of the validity and reliability of the selected survey instrument.
The EQRO should not conduct independent validity and reliability testing of the survey
instrument; however, it should note whether such testing was done. The EQRO should consider
the adequacy of the survey’s reliability and validity testing in determining whether to rely on the
survey findings to inform the EQRO’s analysis and evaluation of access, quality, and timeliness
of health care. See Activity I.3 for more information about assessing the validity and reliability of
survey instruments.

ACTIVITY II.4: REVIEW THE SAMPLING PLAN
WORKSHEET 6.4

The EQRO should assess the sample plan documentation for the following:
1

Clear definition of the study population. The EQRO
should document whether there was a clear
definition of the study population

Resources for Activity 4
Worksheet 6.4. Sampling Plan

Appropriate specifications for the sample frame. The
EQRO should assess whether the sampling frame
was clearly described and appropriate to the survey
objectives

• Provides a set of questions to assess
the sampling plan

Quality of the sampling frame. The EQRO should
assess whether the sampling frame is free from
bias. The sampling frame should include all
members of the population to be studied, and not
omit any members of the population

• Provides an overview of sampling
approaches and guidance for
determining sample sizes for EQR
data collection activities

Type of sampling method used. The EQRO should evaluate whether the sampling method
used was appropriate to the survey’s purpose (e.g., use of probability versus nonprobability methods). For more information, see Appendix B

Adequacy of the sample size. The EQRO should determine whether the sample size was
appropriate for the survey. Two factors influence the determination of the appropriate
sample size for a survey: (1) the acceptable margin of error, and (2) the confidence levels

Procedures for sample selection. The EQRO should review the sample selection
procedures including reviewing the statistical program or other process used to generate
the sample. The EQRO should determine the extent to which the selection of sample
members was conducted to protect against bias

Appendix B. Sampling Approaches for
EQR Data Collection Activities

The level of detail involved in this review requires that the EQRO use professional statisticians.
The EQRO must evaluate whether the sample selected was sufficiently representative of the

PROTOCOL SIX | 243

study population for the EQRO to have confidence in the survey findings. See Activity I.4 for
more information about developing a sampling plan.

ACTIVITY II.5: REVIEW THE ADEQUACY OF THE RESPONSE RATE
WORKSHEET 6.5

In this activity, the EQRO should review the methods
used to maximize the response rate, as well as the
methods used to calculate the response rate. In
addition, the EQRO should assess potential sources of
non-response and bias, and the extent to which the
response rate weakens or strengthens the
generalizability of the survey findings.

The EQRO should determine whether a standard
methodology was used to calculate the response rate.
The American Association of Public Opinion Research (AAPOR) provides a list of standard
definitions and response rate calculators on its website at http://www.aapor.org/StandardsEthics/Standard-Definitions-(1).aspx. To provide context for the assessment of the adequacy of
the response rate, the EQRO should consider benchmarking the response rate against those
achieved by similar surveys. As discussed in Activity I.8, a nonresponse analysis can provide
insights into the representativeness of the survey when response rates are low.
See Activity I.5 for more information on strategies to maximize response.

ACTIVITY II.6: REVIEW THE QUALITY ASSURANCE PLAN
WORKSHEET 6.6

The EQRO should review the quality assurance plan to
ensure that it contains quality checks for all phases of
the data collection effort. Specific areas for focus
include checks during the sampling and locating
processes, customization by data collection mode, and
specification of data quality controls. In addition, the
EQRO should be sure that the plan specifies how the
checks will be implemented. See Activity I.6 for more
information on the quality assurance plan.

244 | PROTOCOL SIX

Resources for Activity 6
Worksheet 6.6. Quality Assurance Plan
• Provides a set of quality check
questions to assess the quality
assurance plan

ACTIVITY II.7: REVIEW THE SURVEY IMPLEMENTATION
WORKSHEET 6.7

The EQRO should review documentation regarding the
survey implementation and assess whether
implementation conformed to the work plan. The EQRO
should specifically consider the following:

Resources for Activity 7
Worksheet 6.7. Survey Implementation
According to the Work Plan
• Provides a set of questions to assess
survey implementation

•

Adherence to the sampling plan

•

How the survey questionnaire was administered,
including formatting and distribution of mailed surveys or scripting and training of telephone
interviewers

•

Changes to the survey schedule

•

Evidence of implementation of the quality assurance checks

•

Problems detected and corrections implemented during the survey process

•

Confidentiality procedures followed

•

Data collection, data entry, and data quality control methods used, including reports of
missing data, data that failed edit checks, and incomplete or unusable surveys

See Activity I.7 for more information on survey implementation.

ACTIVITY II.8: REVIEW THE SURVEY DATA ANALYSIS AND FINAL
REPORT
WORKSHEET 6.8

The EQRO should review how the survey data were
analyzed, including the statistical procedures used and
comparisons made. The EQRO should assess whether
the analysis was appropriate to the survey purpose,
whether appropriate statistical tests were applied, and
how well the survey findings were supported by the data.
In its final validation report, the EQRO should document
its conclusions and provide written findings on:

Resources for Activity 8
Worksheet 6.8. Survey Data Analysis
and Final Report
• Provides a set of questions to assess
the data analysis and final report

•

The survey’s technical strengths and weaknesses

•

Appropriateness of analysis methods (e.g., data quality, sample sizes, weighting and
adjustment for complex sample design if applicable, significance testing)

•

Appropriateness of presentation approaches (such as text, tables, figures)

•

Appropriateness of conclusions drawn from the survey data

•

The limitations and generalizability of survey findings

See Activity I.8 for more information about the presentation of survey findings.

PROTOCOL SIX | 245

TECHNICAL APPENDIX FOR PROTOCOL 6: UNDERSTANDING
POTENTIAL SOURCES OF SURVEY ERROR
Survey results are used increasingly for “high-stakes” activities such as consumer health plan
selection, health plan or provider payment, or performance incentives (e.g., auto-assignment).
As a result, there is increasing scrutiny on the quality and integrity of surveys to support such
initiatives. States and MCPs cannot afford “errors” in a survey, as the consequences may be
substantial from a beneficiary, provider, health plan, and state perspective.
This appendix provides additional information on how to assess the overall quality of the survey
effort using a Total Survey Error (TSE) paradigm. The TSE paradigm identifies potential sources
of survey error and examines the accumulation of all errors that arise in the design, collection,
processing, and analysis of survey data. It is important to note that in the TSE paradigm, errors
are sources of uncertainty, a deviation of a survey response from its underlying true value.
Errors are not mistakes.
The TSE paradigm is included in Figure 6.2. Table 6.3 describes which errors may arise in the
process and steps that EQROs (or survey vendors) can take to remedy the errors. This
information can inform the administration and validation of surveys to improve overall survey
quality.
Figure 6.2. The Total Survey Error Paradigm: Understanding Potential Sources of Error in
Surveys

Source: Adapted from Groves, R.M. et al, Survey Methodology, Hoboken, NJ, John Wiley & Sons, 2004.

246 | PROTOCOL SIX

Table 6.3. Mapping common sources of survey error to data collection activities and
remedies to minimize error

Activity

Common
errors

Definition

Remedies

Specification
error

Sometimes called validity, are we
measuring what we say we are
measuring?

• Use validated scales, pretesting, cognitive
testing, focus groups

Measurement
error

When an answer to a question is
inaccurate, imprecise, or cannot be
compared to other respondents’
answers due to the questionnaire,
respondent, interviewer, or mode

Design survey using best practices:
• Programming checks
• Validated scales
• Multiple languages
• Multiple modes to participate

Coverage
error

When who/the group you want to
study differs from who is available to
study

• Offer multiple modes to participate

Sampling
error

When the survey includes only a
subset of the target population. This
error cannot be avoided unless a
census is conducted

• Conduct power calculations

When people in the survey sample
do not respond and are different from
those who do respond in a way that
is important to the study. There is
Unit nonresponse (sample members
who do not respond to the survey)
and Item nonresponse (sample
members who skip or refuse specific
questions)

• Use proven contact strategies, such as
advance letters and vary modes for
nonresponse follow up

Nonresponse
error

• Use dual frame samples

• Create sampling weights
• Conduct all analyses using weights

• Institute range checks
• Monitor skip patterns and missing data
through frequency reviews
• Conduct critical item retrieval
• Conduct nonresponse bias analysis
• Make nonresponse adjustments

Processing
error

Problems that occur when preparing
“raw” datasets set for analysis, such
as inconsistent coding, treatment of
outliers, or deriving new variables

• Develop cleaning and coding
specifications
• Perform double entry, adjudication, data
review

Adjustment
error

Mistakes in efforts to improve the
quality of the survey estimates as a
result of coverage, sampling, and
non-response errors

• Use post-survey adjustments such as
weighting and imputation

Inferential
error

Making opinionated statements,
drawing incorrect conclusions, or
going beyond the limits of the design

• Prepare comprehensive quality assurance
plans
• Develop rigorous data analysis plans

Source: Adapted from Groves, R.M. et al, Survey Methodology, Hoboken, NJ, John Wiley & Sons, 2004.

END OF PROTOCOL 6

PROTOCOL SIX | 247

WORKSHEETS FOR PROTOCOL 6:
SURVEY ADMINISTRATION AND VALIDATION TOOLS
Instructions. Use these or similar worksheets as a guide when administering or validating a survey. Each numbered
worksheet corresponds to an activity in the protocol. For each question, please check “Yes,” “No,” or “Not applicable.”
If the answer is “No” or “Not applicable,” please explain in the “Comments” column. Add “Comments” for any question
as needed.
•

For survey administration: Use the worksheets to track and document steps performed in designing and
implementing the survey. In the “Comments” column, document decisions or findings

•

For survey validation: Use the worksheets to track and document steps performed in validating the survey. In the
“Comments” column, document the outcome of validation activities, including sources reviewed. The worksheets
can also be used as an outline for the final report to the state. Expand the tool to include other activities or
findings as needed

This tool includes the following worksheets and the applicable activity and step:
Worksheet name

Protocol activity and step

Section I. Administering the Survey
Section II. Validating the Survey
Worksheet 6.1. Survey Purpose,
Objectives, and Audience

Section I. Activity 1. Define the Survey Purpose, Objectives, and
Audience
Section II. Activity 1. Review the Survey Purpose, Objectives, and
Audience

Worksheet 6.2. Work Plan

Section I. Activity 2. Develop the Work Plan
Section II. Activity 2. Review the Work Plan

Worksheet 6.3. Survey Instrument

Section I. Activity 3. Select the Survey Instrument
Section II. Activity 3. Review the Validity and Reliability of the Survey
Instrument

Worksheet 6.4. Sampling Plan

Section I. Activity 4. Develop the Sampling Plan
Section II. Activity 4. Review the Sampling Plan

Worksheet 6.5 Strategy to Maximize
Response

Section I. Activity 5. Develop a Strategy to Maximize Response
Section II. Activity 5. Review the Adequacy of the Response Rate

Worksheet 6.6. Quality Assurance Plan

Section I. Activity 6. Develop a Quality Assurance Plan
Section II. Activity 6. Review the Quality Assurance Plan

Worksheet 6.7. Survey Implementation
According to the Work Plan

Section I. Activity 7. Implement the Survey According to the Work Plan
Section II. Activity 7. Review the Survey Implementation

Worksheet 6.8. Survey Data Analysis
and Final Report

Section I. Activity 8. Prepare and Analyze Survey Data and Present
Results in a Final Report
Section II. Activity 8. Review the Survey Data Analysis and Final Report

248 | PROTOCOL SIX

Worksheet 6.1. Survey Purpose, Objectives, and Audience
Survey purpose, objectives, and audience: ________________________________________________________
Assess the clarity of the survey purpose and audience by answering the following questions. Insert comments to
explain “No” and “Not applicable” responses.

Question

Yes

Not
applicable

Comments

Was there a clear, written statement of the survey
purpose that addresses access, timeliness, and/or
quality of care?
Was the unit of analysis clearly stated?
Did the unit of analysis include individual MCPs?
Was there a clear and measurable written study
objective?
Was the audience for and intended use of the
survey findings identified?
Overall validation assessment: In the comments
section, note any recommendations for improving
the survey purpose, objective, and audience

PROTOCOL SIX | 249

Worksheet 6.2. Work Plan
Date of work plan: ______________________________________________________
Assess the adequacy of the work plan by answering the following questions. Insert comments to explain “No” and
“Not applicable” responses. (Note: Validation of the work plan occurs in conjunction with Activity 5, Review Survey
Implementation According to the Work Plan.)

Question
Did the work plan include a project management
plan (including key staff and roles)?
Did the work plan include a project schedule
(including timelines and deliverable dates)?
Did the work plan specify project reporting
requirements (including the number, format, and
content of the reports)?
• The work plan should include a description of
any reports that the EQRO will be responsible to
publicly release, if this is part of the EQRO’s
scope of work
Did the work plan include a data preparation plan,
such as production of data files, data file format,
and delivery?
Did the work plan include a data analysis plan
(including the use of a statistician as appropriate)?
• The EQRO should use a statistician to develop
an analysis plan that supports the survey
purpose and objectives and is consistent with the
intended use of results
• If feasible, the EQRO should provide the state
with a mock-up of the analysis before
administering the survey. This will assure the
survey analysis will be consistent with the
intended use of results
Did the work plan include data security protocols
and procedures for assuring the confidentiality of
data in compliance with HIPAA?
Overall validation assessment: In the comments
section, note any recommendations for improving
the work plan

250 | PROTOCOL SIX

Yes

Not
applicable

Comments

Worksheet 6.3. Survey Instrument
Name of survey instrument______________________________________________________________________
Assess the selection of the survey instrument by answering the following questions. Insert comments to explain “No”
and “Not applicable” responses. Complete a separate worksheet for each survey instrument.

Question

Yes

Not
applicable

Comments

Was the selected survey instrument appropriate for
the purpose of the survey and the unit of analysis?
Were new items developed for the survey?
If new items were developed, was a test of validity
and reliability conducted for the new items?
Was the overall survey instrument tested for face
validity and content validity and found to be valid?
Was the overall survey instrument tested for
reliability and found to be reliable?
Was testing performed for the specific target
population (e.g., Medicaid or CHIP) and languages?
Overall validation assessment: In the comments
section, note any recommendations for improving
the selection of the survey instrument

PROTOCOL SIX | 251

Worksheet 6.4. Sampling Plan
Assess the sampling plan by answering the following questions. Insert comments to explain “No” and “Not applicable”
responses.

Question
Was the study population clearly defined?
Was the sampling frame clearly defined and
appropriate based on the survey objectives?
Was the sampling frame free from bias?
Was the sampling method appropriate to the survey
purpose?
Was the sample size sufficient for the intended use
of the survey (acceptable margin of error, level of
certainty required)?
Were the procedures used to select the sample
appropriate and protected against bias?
Overall validation assessment: In the comments
section, note any recommendations for improving
the sampling plan

252 | PROTOCOL SIX

Yes

Not
applicable

Comments

Worksheet 6.5. Strategy to Maximize Response
Assess the strategy for locating sample members and specific data needed to administer the survey by answering the
following questions. Insert comments to explain “No” and “Not applicable” responses.

Question

Yes

Not
applicable

Comments

Was locating of sample members conducted to
ensure complete contact information?
Locating is a technique used to improve response
rates by locating and contacting sample members.
This includes verified collection of data, such as first
and last name, home address, email address,
phone number(s), date of birth, language
preference, etc.
Were any of the following strategies included to
maximize response:
• Advance letter
• Multiple and varied call attempts
• Multi-mode surveys
• Multiple languages
Were strategies customized to the study population
(e.g., providers versus beneficiaries)?
Was the method specified for calculating the
response rate, and if so, was the method in
accordance with industry standards?
Was a plan included to conduct a non-response
analysis?
Overall validation assessment: In the comments
section, note any recommendations for improving
the response strategy

PROTOCOL SIX | 253

Worksheet 6.6. Quality Assurance Plan
Date of Quality Assurance Plan: _________________________________________________
Assess the quality assurance plan by indicating whether the following quality checks were included in the plan. Insert
comments to explain “No” and “Not applicable” responses. (Note: The assessment of whether the plan was
implemented appropriately is included in Worksheet 6.7.)

Question
Sampling. Did the plan include a check to ensure
the sample was constructed as specified in the
sampling plan?
Locating. Did the plan include a check that initial
contact was made for every sample member?
Mail data collection. Were the following quality
checks included in the plan?
• Was the survey reviewed for respondent reading
level (surveys should be written at a 6th grade
reading level to ensure most respondents are
able to read and understand the content)
• Were specifications and procedures developed
for formatting, reproducing, and distributing the
survey questionnaire?
• Were contents of the mailing packet, such as the
cover letter and questionnaire, reviewed for
accuracy, print smearing, fading, and
misalignment?
• Were the returned mail surveys data entry
reviewed for accuracy?
Telephone data collection. Were the following
quality checks included in the plan?
• Were interviewer training and telephone scripts
reviewed for accuracy?
• Were telephone interviews monitored to confirm
that interviewers read questions verbatim and
accurately captured responses?
Web-based data collection. Did the plan include a
check that the web-based instrument programming
and content was tested for accuracy?
Data quality controls. Did the plan include
procedures to handle responses that fail edit
checks, treatment of missing data, and
determination of usable/complete surveys? (Note:
The plan should establish a pre-determined number
of questions that must be answered by the
respondent to be considered a usable case.)
Overall validation assessment: In the comments
section, note any recommendations for improving
the quality assurance plan

254 | PROTOCOL SIX

Yes

Not
applicable

Comments

Worksheet 6.7. Survey Implementation According to the Work Plan
Assess the implementation of the survey by answering the following questions. Insert comments to explain “No” and
“Not applicable” responses.

Question

Yes

Not
applicable

Comments

Was the weekly data collection plan implemented
as described in the work plan?
If deviations from the data collection plan occurred,
were the reasons for the deviations explained?
Were quality assurance checks implemented as
specified in the quality assurance plan (see
Worksheet 6.6)? If deviations occurred, please
explain in the Comments column
• Was the sampling plan verified to ensure the
sample was constructed as specified?
• Was initial contact made for every sample
member?
• Were specified quality checks made in
accordance with the data collection mode (mail,
telephone, web-based, or mixed mode)?
• Were procedures developed to handle responses
that fail edit checks, treatment of missing data,
and removal of surveys or data determined to be
unusable?
Overall validation assessment: In the comments
section, note any recommendations for improving
the implementation of the survey

PROTOCOL SIX | 255

Worksheet 6.8. Survey Data Analysis and Final Report
Assess the data analysis and final report by answering the following questions. Insert comments to explain “No” and
“Not applicable” responses.

Question

Yes

Not
applicable

Were post-processing procedures implemented to
address the following:
• Responses that failed edit checks
• Missing data
• Removal of surveys or data determined to be
unusable
Were weights created as appropriate for analyzing
survey responses and generalizing results to the
study population?
Was a nonresponse analysis conducted to
determine if survey respondents differ from
respondents on key variables important to the
findings?
Were survey data analyzed following the analysis
plan laid out in the work plan?
Did the final report include a comprehensive
overview of survey purpose/objective,
implementation, and substantive findings?
Overall validation assessment: In the comments
section, note any recommendations for improving
the data analysis and final report

END OF WORKSHEETS FOR PROTOCOL 6

256 | PROTOCOL SIX

Comments

PROTOCOL 7. CALCULATION OF ADDITIONAL
PERFORMANCE MEASURES
AN OPTIONAL EQR-RELATED ACTIVITY

ACTIVITY 2: CALCULATE MEASURES
ACTIVITY 3: REPORT RESULTS

BACKGROUND
One purpose of quality measurement is to evaluate the degree to which
evidence-based treatment guidelines are followed, where indicated, and
to assess the results of care. The use of quality measurement helps
strengthen accountability and support performance improvement
initiatives at numerous levels. Performance measures can be used to
demonstrate a variety of activities and health care outcomes for particular
populations. For example, states use performance measures to monitor
the performance of individual managed care plans (MCPs) at a point in
time, to track their performance over time, to compare performance
among MCPs, and to inform the selection and evaluation of quality
improvement activities.
Federal regulations at 42 C.F.R. § 438.330(c) require states to specify
standard performance measures for MCPs to include in their
comprehensive quality assessment and performance improvement
(QAPI) programs. 70 Each year, the MCPs must: (1) measure and report to
the state standard performance measures specified by the state; (2)
submit specified data to the state which enables the state to calculate the
standard performance measures; or (3) a combination of these
approaches. Validation of the performance measures specified by the
state for inclusion in MCPs’ QAPI programs is a mandatory external
quality review (EQR)-related activity (see 42 C.F.R. § 438.358(b)(1)(ii)),
as described in Protocol 2. Validation of Performance Measures Reported
by the Managed Care Plan.
Federal regulations at 42 C.F.R. § 438.358(c)(3) specify that the external
quality review organization (EQRO) may calculate performance measures
in addition to those specified by the state for inclusion in MCPs’ QAPI
programs. Calculation of these additional performance measures are an
optional EQR-related activity.

PROTOCOL SEVEN

ACTIVITY 1: PREPARE FOR MEASUREMENT

More information about QAPI and performance measures is available at 42 C.F.R. §
438.330(b)(2). This is cross-referenced by CHIP at 42 C.RF.R 457.1240(b).

257

In many cases, states and MCPs use measures included in the CMS Core Set of Children’s
Health Care Quality Measures for Medicaid and the Children’s Health Insurance Program
(CHIP) (the Child Core Set) and the Core Set of Health Care Quality Measures for Adults
Enrolled in Medicaid (the Adult Core Set). Child and Adult Core Sets to monitor and track quality
of care in Medicaid and the Children’s Health Insurance Program (CHIP). 71 While use of these
measures by states is voluntary, CMS encourages states to adopt and use the Child and Adult
Core Set measures to support their managed care quality measurement and improvement
initiatives. Many Core Set measures are part of the Healthcare Effectiveness Data and
Information Set (HEDIS®), and have national and regional benchmarks.
This protocol provides guidance to states on the calculation of additional (non-QAPI) performance
measures to monitor the care provided by MCPs to enrollees covered by Medicaid and CHIP.

GETTING STARTED ON PROTOCOL 7
Protocol 7 consists of three activities: preparation for measurement, calculation, and reporting
(Figure 7.1). For each activity, the protocol specifies the steps to be performed and the
outcomes to be achieved. The remainder of this protocol outlines the steps associated with
these activities.
Figure 7.1. Protocol 7 Activities

Two supplemental resources are available to help EQROs calculate additional performance
measures:
•

Worksheets for Protocol 7. Performance Measure Calculation Tools, which can be used to
identify the performance measures to be calculated, document the technical specifications
for each measure, develop a master list of data elements, indicate the data sources and
any known data quality issues, and specify the file format for the transmission of the
required data elements

258 | PROTOCOL SEVEN

•

Appendix A. Information System Capabilities Assessment, which is used to assess the
MCP’s data collection, processing, and reporting systems

ACTIVITY 1: PREPARE FOR MEASUREMENT
Step 1: Identify the Performance Measures to be Calculated

WORKSHEET 7.1
WORKSHEET 7.2

The state should provide the EQRO with a list of
performance measures to be calculated along with
technical specifications for their calculation. The EQRO
must understand the state’s specifications for each
performance measure (e.g., sampling guidelines, data
sources, measurement period, instructions for calculating
numerators and denominators), as well as the state’s
requirements for benchmarking, analysis, and reporting.

Resources for Activity 1, Step 1
Worksheet 7.1. List of Performance
Measures to be Calculated
• Template for identifying the measures
the EQRO will calculate for the state,
including the source, how frequently
to calculate each measure, and when
each measure is due to the state

Worksheet 7.2. Companion
The EQRO must also understand the state’s
Performance Measurement Tool
requirements for the timing and format of the
• Template for documenting additional
performance measure report. The EQRO should create a
information about measures in
list of performance measures to be calculated to
Worksheet 7.1, including technical
document the measures required by the state and the
specifications, benchmarks,
reporting frequency and timeline for each measure
performance standards, or other
(Worksheet 7.1). For each performance measure listed
information about state requirements
in Worksheet 7.1, the EQRO should complete a
companion performance measurement worksheet that
contains the technical specifications for the measure, benchmarks, performance standards, and
other information needed to analyze the performance measure according to the state’s
requirements (Worksheet 7.2). The EQRO may need to request clarification from the state,
measure steward, or other expert if the measure specifications are unclear.

Step 2: Prepare for Data Collection

The EQRO should send an introductory communication to the MCP outlining the purpose,
process, and timeline for its performance measure calculation activities. In addition, the EQRO
should request a contact within the MCP to schedule activities and provide requested
documents and other information.

PROTOCOL SEVEN | 259

The EQRO should inform the MCP that it may be necessary to interview MCP or vendor staff
with responsibility for data collection or performance measurement. The information provided by
the MCP should inform the EQRO of the location of the required data, which organization (state,
EQRO, or MCP) will need to collect and integrate specific data elements, and how to access the
data. Information obtained from MCP staff may improve the efficiency and accuracy of the
EQRO’s effort to collect and integrate the data necessary for calculating performance
measures.
During this step, the EQRO should also review or conduct an Information Systems Capabilities
Assessment (ISCA) for each MCP to:
•

Understand data sources, flows, and integration processes used by the MCP

•

Identify where the EQRO needs to work with outside data sources to obtain additional data

•

Determine which data elements are integrated by the MCP and which data elements the
EQRO must integrate

Appendix A contains the ICSA tool and instructions for completing the ISCA. For more
information on how the EQRO should conduct or review an existing ICSA as part of its
performance measurement activities, please refer to Protocol 2, Activity 1, Step 2, Assess the
Integrity of the MCP’s Information System.
If data will be collected from other sources such as state public health registries, vital records,
hospital discharge abstract databases, or behavioral health vendors under contract to the state,
the EQRO should establish contact with the organizations responsible for these data sources.
Step 3: Identify Required Data Elements, Data Sources, and Data Quality Issues

WORKSHEET 7.3
WORKSHEET 7.4

Next, the EQRO should prepare a master list of data
elements (Worksheet 7.3) and identify available data
sources for each required data element, noting any
completeness or integration issues for each element
(Worksheet 7.4).

Resources for Activity 1, Step 3
Worksheet 7.3. Data Element Master
Checklist
• Template for identifying the data
elements needed to calculate each
performance measure

Data sources may include those maintained by an MCP
in a data repository, such as claims or enrollment data.
Data sources may also include sources external to the
Worksheet 7.4. Data Availability and
Data Quality
MCP, such as a state registry, provider medical record,
MCP vendor, or state vendor. The EQRO should
• Template for documenting data
document data capture or integration issues for the
availability and data quality issues
(e.g., completeness and integration)
required data elements, such as an inability to capture
individual prenatal care services when the MCP pays for
maternity care using a global fee. As another example,
the EQRO may identify issues associated with data sources external to the MCP, such as
difficulty accessing confidential information about mental health services that the state contracts

260 | PROTOCOL SEVEN

with another organization to manage, incomplete data in a voluntary state registry, or challenges
in obtaining vital records required for data linkage.

ACTIVITY 2: CALCULATE MEASURES
Step 1: Collect Performance Measure Data

WORKSHEET 7.5

After the required data elements and data sources have
been identified, the EQRO will request the data needed
Resources for Activity 2, Step 1
to calculate the performance measures from the MCP or
Worksheet 7.5. Illustrative File Format
other data suppliers. For each data source, the EQRO
for Transmission of Claims Data
should specify how the data are to be transmitted to the
• Template for constructing an
EQRO, including appropriate privacy and security
electronic data shell or file format
safeguards. To ensure accurate and complete data for
including definitions for all data fields
measure calculation, the EQRO should develop a file
format that specifies the content and structure of the
data file along with definitions of all data fields (Worksheet 7.5).
The EQRO should construct file formats that are customized to each data supplier. The file
format for obtaining data from the MCP data repository will likely include all data elements that
originate from claims/encounter, eligibility, and provider transaction systems. (In some cases,
the data will be available in a state data repository and the file format should reflect the state
system structure.) A file format used to obtain vital records or immunization registry data would
contain different data fields and definitions applicable to those sources.
If the EQRO needs to conduct medical record review, it should develop the following resources:
•

Abstraction tools

•

Training for personnel conducting the medical record abstraction

•

Quality assurance procedures to assess the accuracy and reliability of the medical record
abstraction

•

Electronic data entry edits for abstracted medical record information

If the MCP or other entity is performing medical record review and supplying those data to the
EQRO, the EQRO should refer to Protocol 2 to validate the abstracted medical record
information.
Step 2: Clean Data

As the EQRO receives data, it should evaluate each incoming data stream to ensure that the
number of bits received is equal to the number sent. After entering the data into its repository,
the EQRO should clean the data using electronic edits. Examples of edits include the following:
•

Valid procedure codes (e.g., active code, required number of digits)

•

Valid diagnosis codes (e.g., active code, required number of digits)

PROTOCOL SEVEN | 261

•

Internal consistency of diagnosis and procedure codes (e.g., consistent with the enrollee’s
age or gender, or the practitioner’s specialty)

•

Correct field size and type (e.g., alpha, numeric, date)

•

Valid date ranges (e.g., “to” date is later than “from” date; dates occur during the
appropriate timeframe for the measure)

•

Valid practitioners (e.g., active provider)

•

Valid enrollees (e.g., eligible on date of service)

Data that pass the edit should be integrated in the EQRO’s performance measure repository
(see Step 3). When data fail an edit, the EQRO should contact the supplier and request the data
be corrected and resubmitted. The EQRO should document the nature and extent of failures,
including information about whether it received corrected information. This documentation is
necessary for the EQRO to understand the accuracy and completeness of the data underlying
the performance measures it will calculate.
Step 3: Integrate Data into Performance Measure Repository

The EQRO may receive data from multiple MCPs, multiple sources within each MCP, and other
organizations, such as a statewide registry or other state vendors responsible for delivering
specific benefits like pharmacy or mental health services. To calculate performance measures,
the data must be integrated across the various data sources so that all services provided to a
specific enrollee can be associated with that enrollee. The ISCA, reviewed in Activity 1, Step 3,
will provide information about the adequacy of data integration within each MCP.
During this step, the EQRO will also assess the integration of data from non-MCP data
suppliers, such as the state’s encounter data repository or other vendors. This may include
administering relevant portions of the ISCA to these other suppliers. The EQRO must determine
which portions of the ISCA are relevant depending on the specific data elements the supplier
provides and the degree of data integration the supplier must perform. The assessment includes
assessing the reliability of data transmissions within and from each data supplier. The EQRO
may have different degrees of access to these data suppliers and must work with them, to the
extent possible, to understand the data flows and procedures used to ensure data integrity. For
each data supplier, the EQRO may need to:
•

Examine the details of the data supplier’s processes to accurately and completely transfer
data from the transaction files (i.e., enrollment, provider, encounter/claims) into its data
repository, if any

•

Examine samples of data to assess completeness and accuracy

•

Investigate the data supplier’s processes to consolidate multiple files (sometimes referred
to as deduplicating or “de-duping” of files), and to extract required information from its data
repository

•

Compare actual results of file consolidations or extracts to those that should have resulted
according to documented algorithms or specifications

•

Review procedures for consolidating data from vendors in ways that ensure the accurate,
timely, and complete integration of the data

262 | PROTOCOL SEVEN

•

Review computer program reports or documentation that reflect these vendor coordination
activities, and spot check to verify that no data necessary to performance measure
reporting are lost or inappropriately modified during transfer

•

Assess the extent to which proper linkage mechanisms have been employed to join data
from all necessary sources (e.g., identifying an enrollee with a given disease/condition)

To ensure proper data integration within its own data repository, the EQRO must undertake the
following activities:
•

Write program logic or source code for each measure that identifies, tracks, and links
enrollment within and across product lines (Medicaid and CHIP), by age and sex, as well as
through possible periods of enrollment and disenrollment, which complies with the
specifications of each performance measure

•

Conduct tests of data to assess completeness, integration, and integrity, and to ensure
there is no double-counting of services reported through different data systems or suppliers

•

Assure that all enrollees who were eligible to receive the specified services were included in
the initial population from which the final denominator was produced. The eligible
population will include both enrollees who received the services and those who did not. This
same activity applies to provider groups or other relevant populations identified in the
specifications of each performance measure

Step 4: Conduct Preliminary Analysis

The EQRO will assess the completeness, accuracy, and reasonableness of the data in its
repository, and work with the MCP and other data suppliers until the data are satisfactory.
Referring to the ISCA conducted for the MCP and other data suppliers, the EQRO will have
identified areas of potential weakness. These should be considered in conducting analyses of
missing data, data quality, and supplier data issues.
Missing Data. The EQRO will analyze its repository for evidence of missing data. To determine
completeness, the EQRO should compare its data with data from the state, from prior years,
and from similar populations. Based on findings from the ISCA, the EQRO may pursue specific
concerns such as missing beneficiaries; missing providers, provider locations, or provider types;
and missing services or service types. Knowledge of the data suppliers’ contractual
relationships with providers from the ISCA as well as knowledge of the expected magnitude of
reporting will help identify specific areas to investigate for missing data. The EQRO should be
aware of instances when the MCP was unable to submit data, when submitted data failed edits,
and when data were not resubmitted.
Other Data Quality Issues. The EQRO should analyze the data it has received to identify data
quality problems such as inability to process or retain certain fields. Some MCPs may lack the
capacity to capture or maintain all the data elements that are required for submission, such as
secondary diagnoses or procedure codes or some coding specificity.
Supplier Data Issues. Using the edit checks from Activity 2, Step 2, the EQRO should identify
problems in how data suppliers compiled and submitted their data to the EQRO.

PROTOCOL SEVEN | 263

Significant issues may affect the feasibility of calculating valid and reliable performance
measures. Before proceeding to Step 5, the EQRO should report significant issues to the state
and determine whether the data are suitable for calculating selected measures.
Step 5: Calculate the Denominators, Numerators, and Rates

Following the specifications provided by the state, the EQRO calculates performance measures
from its data repository. To do this, the EQRO must purchase or write and test source code to
properly apply all specifications to identify the denominator population. The EQRO must apply
specified continuous enrollment and other eligibility criteria and implement exclusions from the
denominator.
Once the EQRO has identified all eligible beneficiaries in the denominator for a measure, it must
apply the specifications to identify cases that qualify for inclusion in the numerator. Where
sampling is required such as for medical record review, the EQRO must follow the specifications
for selecting an appropriate sample. The EQRO should follow the medical record review
process outlined in Activity 2, Step 1 regardless of when in the measure calculation process the
medical record review takes place.

ACTIVITY 3: REPORT RESULTS
Step 1: Report Preliminary Performance Measure Results

Before sharing performance measure results with the state, the EQRO should share its
preliminary findings with the MCPs to obtain their feedback about the accuracy of the results.
The report should include, at a minimum, the following elements for each performance measure:
•

Data source(s)

•

Method (administrative, medical record review, hybrid)

•

Denominator

•

Sample size (if relevant)

•

Administrative numerator events (if relevant)

•

Medical record numerator events (if relevant)

•

Calculated rate

•

Deviations from the measure specifications (if relevant)

To enable the MCP to understand and interpret the results, the report may also analyze MCP
performance in relation to external benchmarks or prior-year performance.
The EQRO should invite the MCP to offer comments and documentation to support correction of
any factual errors or to clarify results. The EQRO should provide a reasonable period of time for
the MCP to provide its comments. The EQRO should then recalculate measures based on the
comments, if necessary, and revise its findings where appropriate.

264 | PROTOCOL SEVEN

Step 2: Analyze Performance Measure Results

Using the final calculations from Activity 3 Step 1, the EQRO should conduct all analyses
required by the state. The results should be presented in a format prescribed by the state, or if
the state has not prescribed a format, in a way that facilitates the state’s intended use of the
performance measures. Decisions about the format include the balance between text, tables,
and graphics, as well as the level of detail. In addition, the EQRO should determine whether the
data should be analyzed for each MCP individually, or whether individual MCP results should
also be compared to results for other MCPs or in relation to external benchmarks.
Step 3: Submit a Final Report to the State

The EQRO will submit a final report containing the performance measure results, analyses, and
recommendations in the format prescribed by the state and in the time frame required. The
content of the final report should include the following elements:
•

A summary of the EQRO’s performance measurement activities, including documentation
of the activities performed

•

Work papers and detailed results of key steps of the measure calculation process, MCPspecific performance measure rates, and accompanying analyses

•

Discussion of areas of MCP strength and opportunities for improvement in both data
management and performance

•

Recommendations for improving MCP performance

The EQRO should submit a draft report on performance measure results to the state (separate
from the EQR technical report) and revise the final report based on feedback from the state.
END OF PROTOCOL 7

PROTOCOL SEVEN | 265

WORKSHEETS FOR PROTOCOL 7:
PERFORMANCE MEASURE CALCULATION TOOLS
Instructions. Use these or similar worksheets to identify the performance measures to be calculated, document the
technical specifications for each measure, develop a master list of data elements, indicate the data source and any
known data quality issues, and specify the file format for the transmission of the required data elements. This tool
includes the following worksheets crosswalked to the applicable Activity and Step:
Worksheet name

Protocol activity and step

Worksheet 7.1. List of Performance
Measures to be Calculated

Activity 1. Step 1. Define the Scope of the Validation

Worksheet 7.2. Companion Performance
Measurement Tool

Activity 1. Step. 1. Define the Scope of the Validation
Activity 1. Step 3. Conduct Detailed Review of Measures
Activity 2. Step 4. Complete the Detailed Review of Measures

Worksheet 7.3. Data Element Master
Checklist

Activity 1. Step 4. Initiate Review of Medical Record Data Collection
Activity 3. Post-Site Visit Activities

Worksheet 7.4. Data Availability and Data
Quality

Activity 1. Step 5. Prepare for the MCP Onsite Visit

Worksheet 7.5. Illustrative File Format for
Transmission of Claims Data

Activity 2. Step 1. Review Information Systems Underlying
Performance Measurement

□ Staff model □ Network □ IPA

Plan type

□ MCO □ PIHP □ PAHP □ PCCM □ LTSS
□ Other: specify ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid
and CHIP

Note: IPA = Independent Practice Association; LTSS = Long-Term Services and Supports; MCO = Managed Care Organization;
PIHP = Prepaid Inpatient Health Plan; PCCM = Primary Case Management.

266 | PROTOCOL SEVEN

Worksheet 7.1. List of Performance Measures to be Calculated
This worksheet is used to identify the measures to be calculated, including the measure source, how frequently the
measure is reported, and the reporting deadline. Complete the worksheet for each measure to be calculated, and
adapt as needed. Please note if you use the HEDIS® or Child/Adult Core Set specifications. Also note if the measure
is homegrown.
NQF # (if
applicable)

Measure
Steward

Performance measure

Measure source

Reporting
frequency

Reporting
deadline

The list below is illustrative of the information to be included in the worksheet. Please include National Quality Forum
(NQF) number (if applicable), and measure steward.
NQF # (if
applicable)

Measure
Steward

0038

NCQA

Childhood Immunization Status
(CIS-CH)

HEDIS® 2017/
Child Core Set

Annual

June 30th

1407

NCQA

Immunizations for Adolescents
(IMA-CH)

HEDIS® 2017/
Child Core Set

Annual

June 30th

1392

NCQA

Well-Child Visits in the First 15
Months of Life (W15-CH)

HEDIS® 2017/
Child Core Set

Annual

June 30th

1516

NCQA

Well-Child Visits in the Third, Fourth,
Fifth and Sixth Years of Life (W34CH)

HEDIS® 2017/
Child Core Set

Annual

June 30th

Performance measure

Measure source

Reporting
frequency

Reporting
deadline

PROTOCOL SEVEN | 267

Worksheet 7.2. Companion Performance Measurement Tool
This worksheet is used to document the technical specifications and benchmarks for each measure to be calculated.
Complete the worksheet for each measure listed in Worksheet 7.1, and adapt as needed.
Measure Name and Description __________________________________________________________________
Characteristics

Measure specifications

Measure purpose (check all that
apply)







QI or PIP
Demonstration/waiver program (e.g., 1115 demonstration)
Pay for performance/Value-based purchasing
Public reporting
Other (specify) ____________________

Data collection method (check
one)







Administrative
Medical Record Review
Hybrid (administrative supplemented by medical record review)
Survey
Other (specify) ____________________

Sampling method (if applicable)

Specifications for sample size, sampling method and replacement methods:
____________________________________________________________

Age

Lower age limit: ____________________
Upper age limit: ____________________

Sex (check one)

 Males only
 Females only
 Males and females

Continuous enrollment

 No
 Yes (specify) ________________________

Index event (e.g., birthday;
discharge; prescription;
diagnosis; procedure)

 No
 Yes (specify) ________________________

Denominator elements and data
sources (e.g., member ID, age,
gender, enrollment and
disenrollment dates, diagnoses,
procedures)

A list of each data element needed to establish eligibility for the denominator:
_____________________________________________
For each denominator element, the allowable data source(s):
_____________________________________________

Numerator elements and data
sources (e.g., procedure codes,
diagnosis codes, pharmacy
codes, lab results, dates of
service)

A list of each data element needed to establish eligibility for the numerator:
_____________________________________________
For each numerator element, the allowable data source(s):
_____________________________________________

Denominator

Denominator statement:
_______________________________________________________________
Inclusions/exclusions:
_______________________________________________________________
Denominator time window:
_______________________________________________________________

Numerator

Numerator statement:
_______________________________________________________________
Inclusions/exclusions:
_______________________________________________________________
Numerator time window:
_______________________________________________________________

268 | PROTOCOL SEVEN

Characteristics

Measure specifications

Rate calculation

Formula for calculation of rate:
_________________________________________________________

Benchmark(s) (check all that
apply)

 State-level (specify):
____________________________________________________________
 Regional (specify):
____________________________________________________________
 National (specify):
____________________________________________________________
 Other (specify):
____________________________________________________________
Source(s):
____________________________________________________________

Other analysis requirements
(e.g., change from prior year or
comparison to state average or
best in state, including
statistical tests)

List required analyses:
_____________________________________________________________

PROTOCOL SEVEN | 269

Worksheet 7.3. Data Element Master Checklist
This checklist is used to develop a master list of data elements needed to calculate each performance measure.
Indicate whether each data element is required to calculate the measure, and adapt as needed.
Performance
Measure 1
Denominator
data elements:
Date of birth
Sex
Enrollment date
Disenrollment
date
Diagnosis code
Procedure code
Service date
Provider ID
Numerator data
elements:
Diagnosis code
Procedure code
Pharmacy code
Lab order
Lab result

270 | PROTOCOL SEVEN

Performance
Measure 2

Performance
Measure 3

Performance
Measure 4

Performance
Measure 5

Worksheet 7.4. Data Availability and Data Quality
This worksheet is used to document the data availability and data quality issues for each data element required to
calculate the denominator and numerator for each measure in Worksheet 7.1.
Available data
source(s)?

In data
repository?

Identified data quality issues (e.g.,
data completeness, integration issues)

Denominator data elements:
Date of birth

 Yes
 No

Sex

 Yes
 No

Enrollment date

 Yes
 No

Disenrollment date

 Yes
 No

Diagnosis code

 Yes
 No

Procedure code

 Yes
 No

Service date

 Yes
 No

Provider ID

 Yes
 No

[INSERT other denominator
data elements]

 Yes
 No

Numerator data elements:
Diagnosis code

 Yes
 No

Procedure code

 Yes
 No

Pharmacy code

 Yes
 No

Lab order

 Yes
 No

Lab result

 Yes
 No

[INSERT other numerator data
elements]

 Yes
 No

PROTOCOL SEVEN | 271

Worksheet 7.5. Illustrative File Format for Transmission of Claims Data
This worksheet provides a template for constructing an electronic data shell or file format, including definitions for all
data fields. Complete this template for each data field in the claims file, and adapt as needed.
Field
#

Data Field

Applies to
UB Phys Rx

Type/Format

Req/Opt

Comments

Row Type

Char(1)

Required

1=UB, 2=Phys, 3=Rx

Claim Status

Char(1)

Required

P=Paid, D=Denied
Denied claims are
highly desirable for
accurate performance
measurement

Recipient ID

Varchar(50)

Required

Medicaid or CHIP
identifier supplied by
the State for the
member. Native or
encrypted. If encrypted,
separate encryption key
must be provided.

Claim Number

Varchar(80)

Required

Required if source is
not sending final-only
versions of claims

Prior Version Claim
Number

Varchar(80)

Required

Required if source is
not sending final-only
versions of claims

Claim Received
Date

yyyymmdd

Required

Required if source is
not sending final-only
versions of claims

Claim Paid Date

yyyymmdd

Required

Required if source is
not sending final-only
versions of claims

Billing Provider ID

Varchar(30)

Required

Any internal identifier
for the billing provider.
Must be unique to one
clinician or entity. Must
exist on the provider
file. If supplying for Rx,
use pharmacy provider
ID.

Principal Diagnosis

Varchar(5)

Required

No periods, left justified

Diagnosis 2

Varchar(5)

Required

No periods, left justified

Diagnosis 3

Varchar(5)

Required

No periods, left justified

END OF WORKSHEETS FOR PROTOCOL 7

272 | PROTOCOL SEVEN

PROTOCOL 8. IMPLEMENTATION OF
ADDITIONAL PERFORMANCE IMPROVEMENT
PROJECTS
AN OPTIONAL EQR-RELATED ACTIVITY

ACTIVITY 2: DEFINE THE PIP AIM STATEMENT
ACTIVITY 3: IDENTIFY THE PIP POPULATION
ACTIVITY 4: USE SOUND SAMPLING METHODS
ACTIVITY 5: SELECT THE PIP VARIABLES
ACTIVITY 6: COLLECT VALID AND RELIABLE DATA
ACTIVITY 7: ANALYZE DATA AND INTERPRET RESULTS
ACTIVITY 8: REVIEW IMPROVEMENT STRATEGIES
ACTIVITY 9: ASSESS WHETHER SIGNIFICANT AND SUSTAINED
IMPROVEMENT OCCURRED

BACKGROUND
Federal regulations at 42 C.F.R. § 438.330(b)(1) and 457.1240(b) require
that Medicaid and CHIP managed care plans (MCPs) conduct
performance improvement projects (PIPs) that focus on both clinical and
nonclinical areas as part of a comprehensive quality assessment and
performance improvement (QAPI) program. 72 Validation of the PIPs
conducted by MCPs as a part of their QAPI programs is a mandatory
external quality review (EQR)-related activity, as described in Protocol 1.
Validation of Performance Improvement Projects.
In addition, Federal regulations at 42 C.F.R. § 438.358(c)(4) and
457.1250(a) specify that the external quality review organization (EQRO)
may conduct PIPs in addition to those performed by the MCPs as a part
of their QAPI programs. These additional PIPs are an optional EQRrelated activity.
This protocol provides guidance to states on the implementation of
additional (non-QAPI) PIPs to assess and improve processes and
outcomes of care provided by MCPs in the state.

PROTOCOL EIGHT

ACTIVITY 1: SELECT THE PIP TOPIC

At a minimum, a single PIP that focuses on both clinical and non-clinical aspects of care may
satisfy this requirement. Otherwise, a state must require at least two PIPs, one clinical and one
non-clinical.

273

GETTING STARTED ON PROTOCOL 8
Protocol 8 consists of nine activities for implementing additional PIPs (Figure 8.1).
Figure 8.1. Protocol 8 Activities

274 | PROTOCOL EIGHT

As shown in Table 8.1, these activities align with the nine steps in Activity 1, Protocol 1,
Validation of Performance Improvement Projects. To streamline the content in this protocol, and
avoid duplication with Protocol 1, please refer to the relevant sections in Protocol 1 and the
associated worksheets.
Table 8.1. Crosswalk between Protocol 8, PIP Implementation and Protocol 1, PIP
Validation
Protocol 8. PIP Implementation
Activities

Protocol 1. PIP Validation
Activity 1 Steps

Protocol 1. PIP Validation
Tool Worksheets

Activity 1. Select the PIP Topic

Step 1. Review the Selected PIP Topic

Worksheet 1.1

Activity 2. Define the PIP Question

Step 2. Review the PIP Aim Statement

Worksheet 1.2

Activity 3. Identify the PIP
Population

Step 3. Review the Identified PIP Population

Worksheet 1.3

Activity 4. Use Sound Sampling
Methods

Step 4. Review the Sampling Method

Worksheet 1.4

Activity 5. Select the PIP Variables

Step 5. Review the Selected PIP Variables
and Performance Measures

Worksheet 1.5

Activity 6. Collect Valid and Reliable
Data

Step 6. Review the Data Collection
Procedures

Worksheet 1.6

Activity 7. Analyze Data and
Interpret Results

Step 7. Review Data Analysis and
Interpretation of PIP Results

Worksheet 1.7

Activity 8. Review Improvement
Strategies

Step 8. Assess the Improvement Strategies

Worksheet 1.8

Activity 9. Assess Whether
Significant and Sustained
Improvement Occurred

Step 9. Assess the Likelihood that
Significant and Sustained Improvement
Occurred

Worksheet 1.9

Three supplemental resources are available to help EQROs design and implement additional
PIPs which can lead to significant and sustained improvement in health care delivery processes
and outcomes:
•

Protocol 1. Validation of Performance Improvement Projects

•

Worksheets for Protocol 1. PIP Validation Tools and Reporting Framework

•

Appendix B. Sampling Approaches for EQR Data Collection Activities

ACTIVITY 1: SELECT THE PIP TOPIC
Additional PIP topics should target improvement in clinical and/or nonclinical services
provided to Medicaid and CHIP enrollees in the state. Selected topics should reflect the
characteristics of Medicaid/CHIP enrollees in terms of demographics, prevalence of disease,
and the potential consequences of the disease. It is recommended that PIP topics align
with: (1) the National Quality Strategy,73 (2) the CMS Quality Strategy,74 and (3) the state’s
managed care quality strategy. The EQRO should also review the state’s performance on the
More information about the HHS National Quality Strategy is available at
https://www.ahrq.gov/workingforquality/about/index.html.

More information about the CMS Quality Strategy is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityInitiativesGenInfo/CMS-Quality-Strategy.html.

PROTOCOL EIGHT | 275

CMS Core Set of Children’s Health Care Quality Measures for Medicaid and the Children’s
Health Insurance Program (CHIP) (the Child Core Set) and the Core Set of Health Care Quality
Measures for Adults Enrolled in Medicaid (the Adult Core Set) to identify measures whose
performance could be impacted by a managed care PIP.75 Additional PIPs can be used to help
drive improvement on these measures. For more information about selecting a PIP topic, see
Protocol 1, Activity 1, Step 1.

ACTIVITY 2: DEFINE THE PIP AIM STATEMENT
The PIP aim statement identifies the focus of the PIP and establishes the framework for data
collection and analysis. The PIP aim statment should define the improvement strategy,
population, and time period. It should be clear, concise, and answerable. When identifying the
PIP aim statement, potential sources of information include:
•

State performance on the Child and Adult Core Set

•

State data relevant to the topic being studied

•

MCP data relevant to the topic being studied

•

Enrollee focus groups or surveys

•

Relevant clinical literature on recommended care and external benchmarks

For more information about developing the PIP aim statement, see Protocol 1, Activity 1, Step 2.

ACTIVITY 3: IDENTIFY THE PIP POPULATION
The additional PIP must clearly identify the target population in relation to the PIP aim statement
(such as age, length of enrollment, diagnoses, procedures, and other characteristics).
Depending on the nature of the PIP aim statement, PIP population, and available data, the PIP
may include the entire population or a sample of the population. Studies that rely on existing
administrative data, such as claims and encounter data, registry data, or vital records are
typically based on the universe of the PIP's population. PIPs that require medical record review
typically include a representative sample of the identified population. If a sample is used, go to
Activity 4. If the entire population will be studied, skip Activity 4 and go to Activity 5. If HEDIS®
measures and sampling methodology are used, go to Activity 5. For more information about
identifying the PIP population, see Protocol 1, Activity 1, Step 3.

ACTIVITY 4: USE SOUND SAMPLING METHODS
Appropriate sampling methods are necessary to ensure the collection of information that
produces valid and reliable results. Refer to Appendix B, Sampling Approaches for EQR Data
Collection Activities, for an overview of sampling methodologies applicable to PIPs. When
HEDIS® measures are used and sampling is required (e.g., for measures calculated using the
hybrid method), HEDIS® sampling methodology should be used. For more information about
sampling approaches for EQR activities, see Protocol 1, Activity 1, Step 4, and Appendix B.

276 | PROTOCOL EIGHT

ACTIVITY 5: SELECT THE PIP VARIABLES
The next step is to select the PIP variables. Variables can take a variety of forms as long as the
selected variables measure performance on the PIP aim statement objectively and reliably and
use clearly defined indicators of performance. The additional PIP should include the number
and type of PIP variables that are adequate to answer the PIP aim statement and for which
appropriate and reliable data are available to measure performance and track improvement
over time. Data availability should also be considered when selecting PIP variables, as more
frequent access to data, such as on a monthly, quarterly, or semi-annual basis, supports
continuous quality improvement (QI) and Plan Do Study Act (PDSA) efforts and can allow and
MCP or state to correct or revise course more quickly, if needed. CMS encourages states to
select PIP variables and performance measures that can be examined on at least a semiannual basis.
To the extent possible, CMS encourages EQROs to choose variables for PIPs that reflect health
outcomes. Performance measures are then used to measure these outcomes. When selecting
measures for an additional PIP, first consider existing performance measures because the
specifications for these measures often have been refined over time, may reflect current clinical
guidance, and may have benchmarks for assessing performance. CMS encourages the use of
the following existing performance measure sets: Child and Adult Core Set measures,
behavioral health clinic quality measures, and Core Quality Measures Collaborative.76 Other
examples of existing measures include NCQA’s Healthcare Effectiveness Data Information Set
(HEDIS®) or measures that have been developed by AHRQ (such as the prevention quality
indicators, inpatient quality indicators, patient safety indicators, and pediatric quality
indicators).77
When there are gaps in existing measures, new measures may need to be developed based on
current clinical practice guidelines or health services research. Consider the following questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study
question?

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the measure to be calculated reliably and accurately?
Are there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

For more information about the selection of PIP variables, see Protocol 1, Activity 1, Step 5.

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about measures for behavioral health clinics is available at https://www.samhsa.gov/section-223/quality-measures. More
information about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-InitiativesPatient-Assessment-Instruments/QualityMeasures/Core-Measures.html.
77
More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.
76

PROTOCOL EIGHT | 277

ACTIVITY 6: COLLECT VALID AND RELIABLE DATA
Data collection procedures during implementation of the PIP must ensure that the data used to
measure performance are valid and reliable. Validity means that the data are measuring what is
intended to be measured. Reliability means that the data are producing consistent results.
To ensure validity and reliability of the data collected as part of implementing the PIP, the data
collection plan should specify:
•

The data sources for the PIP

•

The data to be collected

•

How and when the data are to be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments used to collect the data

The PIP may involve two main kinds of data collection: administrative data sources and medical
record review. Procedures to collect data from administrative data systems will be different from
procedures for visual inspection of medical records or other primary source documents.
However, both types of data collection require assurances that data are valid and reliable. For
more information about assuring the validity and reliability of PIP data collection procedures,
see Protocol 1, Activity 1, Step 6.

ACTIVITY 7: ANALYZE DATA AND INTERPRET RESULTS
Data analysis begins with assessing performance on the selected clinical or nonclinical
measures using appropriate statistical techniques, as specified in the data analysis plan.
Interpretation and analysis of the PIP data should be based on a continuous improvement
philosophy and reflect an understanding of lessons learned and opportunities for improvement.
Interpretation of the PIP results should involve assessing the causes of less-than-optimal
performance and collecting data to support the assessment. Accurate data analysis, including
measurements at multiple points in time and tests for statistical significance, is essential
because the state or MCP may implement changes based on the results. For more information
on data analysis and interpretation of PIP results, see Protocol 1, Activity 1, Step 7.

ACTIVITY 8: REVIEW IMPROVEMENT STRATEGIES
Building on the data analysis and interpretation of PIP results in Activity 7, the next step is to
review the improvement strategies implemented as part of the PIP. Significant, sustained
improvement is the result of developing and implementing effective improvement strategies
(including strategies that are culturally and linguistically appropriate for the target population).
Selected strategies should be evidence-based, that is, there should be existing evidence
(published or unpublished) suggesting that the test of change would be likely to lead to the
desired improvement in processes or outcomes (as measured by the PIP variables). The
effectiveness of the improvement strategy is determined by measuring change in performance
according to predefined measures.

278 | PROTOCOL EIGHT

CMS encourages states to work with their EQROs and MCPs to determine which PIP
methodology best suits the needs of the state, its MCPs, and their beneficiaries. For example, a
state may use the Institute for Healthcare Improvement’s (IHI) Model for Improvement to guide
improvement work and test changes on a small scale using Plan Do Study Act (PDSA) cycles. 78
PDSA cycles provide a methodology to test changes on a small scale and to apply rapid-cycle
learning principles to adjust intervention strategies over the course of the improvement. This
approach involves a continuous cycle of measuring and analyzing performance and requires
frequent reflection and course correction. Data should be evaluated on a regular basis and
interventions should be adjusted based on what was learned. Interventions can then be scaled
to larger settings or populations if found effective. PIPs, based on the Model for Improvement
and PDSA process are sometimes known as rapid-cycle PIPs. For more information on the use
of the IHI Model for Improvement and PDSA cycles, see Protocol 1, Activity 1, Step 8.

ACTIVITY 9: ASSESS WHETHER SIGNIFICANT AND SUSTAINED
IMPROVEMENT OCCURRED
A PIP is intended to result in significant and sustained improvement in health care delivery
processes and outcomes, rather than short-term or random change. The final activity in the PIP
is to assess whether the PIP resulted in statistically significant changes over time that could
reasonably be attributed to the improvement strategy implemented as part of the PIP.
To assess whether significant and sustained improvement occurred, repeated measurements
are required, using the same methodology used for the baseline measurement. In addition, tests
of statistical significance are required to assess whether there is evidence of statistically
significant improvement. For more information on assessing the likelihood that significant and
sustained improvement occurred, see Protocol 1, Activity 1, Step 9.
END OF PROTOCOL 8

More information about the Model for Improvement and PDSA approach is available from the following sources:
http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx and
https://innovations.ahrq.gov/qualitytools/plan-do-study-act-pdsa-cycle.

PROTOCOL EIGHT | 279

This page left blank for double-sided copying.

280 | PROTOCOL EIGHT

PROTOCOL 9. CONDUCTING FOCUS STUDIES
OF HEALTH CARE QUALITY
AN OPTIONAL EQR-RELATED ACTIVITY

ACTIVITY 2: DEFINE THE STUDY QUESTION(S)
ACTIVITY 3: SELECT THE STUDY VARIABLE(S)
ACTIVITY 4: DEVELOP A PLAN TO STUDY THE POPULATION
ACTIVITY 5: COLLECT DATA
ACTIVITY 6: ANALYZE AND INTERPRET STUDY RESULTS
ACTIVITY 7: REPORT RESULTS TO THE STATE

BACKGROUND
States may direct their external quality
How does a Focus Study
review organizations (EQROs) to
Differ from a PIP?
conduct focus studies for quality
A Focus Study is a study of a
improvement (QI), administrative,
particular aspect of clinical care or
legislative, or other purposes. Similar to
nonclinical services provided by
performance improvement projects
an MCP at a point in time.
(PIPs), focus studies may examine
A PIP is a project that implements
clinical or nonclinical aspects of care
an intervention designed to
provided by managed care plans
achieve and sustain significant
(MCPs). Focus studies assess quality of
improvement over time.
care at a point in time, whereas PIPs
assess improvement over time (See box,
How does a Focus Study Differ from a PIP?). For example, a focus study
may be conducted for a single year to provide the state with information
about the baseline status of health care quality for a particular aspect of
care across managed care in the state or for subpopulations served by
managed care within the state. By comparison, PIPs are designed to
achieve significant improvement, sustained over time, in health outcomes
and enrollee experience and include the implementation of interventions
to achieve improvement, evaluation of the intervention’s effectiveness
(including performance measurement), and initiation of activities to
increase or sustain improvement.

PROTOCOL NINE

ACTIVITY 1: SELECT THE STUDY TOPIC(S)

281

Although the goals and regulations for focus studies and PIPs differ, EQROs can use similar
processes to design both types of projects. Both must be designed, conducted, and reported in
a methodologically sound manner. Because of these similarities, the process for conducting
focus studies described in this protocol mirrors many of the activities in Protocol 8 for
conducting PIPs.

GETTING STARTED ON PROTOCOL 9
To complete this protocol, the EQRO undertakes seven activities for each MCP (Figure 9.1).
Figure 9.1. Overview of Protocol 9 activities

282 | PROTOCOL NINE

ACTIVITY 1: SELECT THE STUDY TOPIC(S)
Focus studies may target a single MCP, a subset of MCPs, or all MCPs in the state. They
should target relevant areas of clinical care and nonclinical services in which it is known or
suspected that improvement is needed. For example, the focus study may examine patterns of
over- or under-utilization of services to assess the potential threat to health or functional status
of enrollees. Selected topics should:
•

Reflect MCP enrollment in terms of demographic characteristics, prevalence of disease,
and the potential consequences of the disease

•

Affect a significant portion of enrollees, a specified subpopulation of enrollees, or a
significant portion of enrollees impacted by a specific health care issue (such as oral health
or maternal and infant health)

•

Align with priority areas as identified in the HHS and/or CMS quality strategies. 79

When selecting the focus study topic, the EQRO and state should consider a variety of factors
related to enrollee characteristics, health risks, experience of care, and special population or
service needs (See box, Factors to Consider when Selecting a Focus Study Topic).
Factors to Consider when Selecting a Focus Study Topic
• Demographic and epidemiologic information about current MCP enrollees
• Enrollee health risks and disease prevalence
• State performance on CMS Child and Adult Core Set measures
• Input from enrollees about specific services, such as mental health or substance abuse
• A spectrum of enrollee populations and services:
○ Care for children with special health care needs
○ Care for adults with physical disabilities
○ Care for people with intellectual and developmental disabilities
○ Care for people with dual eligibility who use long-term services and supports (LTSS)
○ Preventive care
○ Acute and chronic care
○ High-volume and high-risk services (even if they are low frequency)
○ Specialized care received from centers (burn, transplant, and cardiac surgery centers)
○ Continuity or coordination of care from multiple providers and over multiple episodes
○ Appeals and grievances
○ Access to and availability of care

79
More information about the HHS Quality Strategy is available at https://www.ahrq.gov/workingforquality/about/index.html.
More information about the CMS Quality Strategy is available at https://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityInitiativesGenInfo/CMS-Quality-Strategy.html.

PROTOCOL NINE | 283

ACTIVITY 2: DEFINE THE STUDY QUESTION(S)
In this activity, the EQRO defines the study question(s). The study question identifies the focus
of the study and establishes the framework for data collection and analysis. The study question
should be clear, concise, and answerable (See Q&A box below). Table 9.1 critiques illustrative
study questions for a focus study.
Q: How do we know if a focus study question is clear, concise, and answerable?
A: If the question specifies measureable indicators and analytics for a defined population and time period.
Tip: Potential sources of information to help form the study question include:
○ State data relevant to the topic being studied
○ MCP data relevant to the topic being studied
○ CMS Child and Adult Core Set performance measures
○ Enrollee focus groups or surveys
○ Relevant clinical literature on recommended care and external benchmarks

Table 9.1. Examples of focus study questions
Illustrative study questions
Poor
Study
Question

What is the status of preventive dental care in
Medicaid?

Good
Study
Question

How does the rate of preventive dental visits
among children enrolled in Medicaid for at least
six months in calendar year 2016 vary by age,
geographic location, and race/ethnicity?

Critique
• Does not specify measurable indicators and
analytics
• Does not define the population and time period
• Specifies the measurable indicator (preventive
dental visits)
• Specifies the analytic issue of interest (variation
in utilization rates by age, geographic location,
and race/ethnicity)
• Defines the population and time period
(children enrolled in Medicaid for at least six
months in calendar year 2016)

ACTIVITY 3: SELECT THE STUDY VARIABLE(S)
In this activity, the EQRO selects the study variable(s)
(See box, What is a Study Variable?). Study variables can
take a variety of forms as long as the selected variables
identify the MCP’s performance on the study questions
objectively and reliably and use clearly defined
measurable indicators of performance. The study
variables for the focus study should allow the EQRO to
measure the MCP’s performance on the elements of care
identified in the study question(s). For example, for a focus
study on preventive dental services in an MCP, the EQRO
should select one or more study variables that directly
assess enrollees’ access to and use of these services.
Examples of such study variables may include:

284 | PROTOCOL NINE

What is a Study Variable?
A study variable is a measurable
characteristic, quality, trait, or attribute
of a particular individual, object, or
situation being studied.
Tips for Choosing Study
Variables
When choosing study variables,
consider different types of variables
and choose the variables that are best
suited to the available data, resources,
and study questions.

•

The proportion of eligible enrollees who received any preventive dental service within a
defined timeframe

•

The proportion of eligible enrollees who received a dental sealant within a defined
timeframe

•

The ratio of dental service providers providing preventive services per 1,000 MCP enrollees
within a defined geographic area

•

The proportion of MCP enrollees who report being able to obtain preventive dental services
within a specified timeframe in an enrollee survey

The EQRO should choose the number and type of study variables that are adequate to answer
the study question(s) and for which appropriate and reliable data are available to determine the
state of the population at a point in time. Study variables may be continuous, categorical, or
discrete (Table 9.2), and use a variety of measurement scales to assess performance (Table
9.3).
Table 9.2. Types of variables for focus studies
Variable type

Definition

Example

Continuous

Have a range of numerical values
Note: Data collected for a continuous variable can
be recoded as a discrete variable (e.g., an enrollee’s
blood pressure is above or below a specified level)

• Age, blood pressure, temperature,
height/weight, body mass index,
birthweight

Categorical

Have a range of non-ordered, qualitative values (or
categories)

• An enrollee survey question that asks
enrollees to identify the most
important among a list of incentives
offered to improve well-care visit rates

Discrete

Have a limited number of possible categories
Note: binary variables have two categories

• An enrollee has/has not received a flu
shot in the past 12 months

Table 9.3. Types of measurement scales for focus studies
Measurement
scales

Definition

Example

Interval

The distances between numbers denote significant
and interpretable differences (e.g., dollars, degrees,
inches, pounds) and the differences are
interpretable as higher or lower.

• The interval between an annual
income of $40,000 and $30,000 =
$10,000

Ordinal

Can be treated as quantitative in some
circumstances, and qualitative in others

• An enrollee survey question that asks
enrollees to rank their experience of
care on a scale from 1 (low quality) to
5 (high quality)

Nominal

The set of categories for a qualitative variable

• Mode of transportation to work (car,
bus, subway, bicycle, walk)

When selecting study variables, the EQRO should consider measures that currently exist within
the health services research community or the managed care industry because the
specifications for these measures often have been refined over time, may reflect current clinical
guidance, and may have benchmarks for assessing MCP performance. CMS encourages use of
the CMS Core Set of Children’s Health Care Quality Measures for Medicaid and the Children’s

PROTOCOL NINE | 285

Health Insurance Program (CHIP) (the Child Core Set) and the Core Set of Health Care Quality
Measures for Adults Enrolled in Medicaid (the Adult Core Set), behavioral health clinic quality
measures, and Core Quality Measures Collaborative for examples. 80 Additional examples of
existing measures include NCQA’s Healthcare Effectiveness Data Information Set (HEDIS®) or
measures that have been developed by AHRQ (such as the prevention quality indicators,
inpatient quality indicators, patient safety indicators, and pediatric quality indicators). 81
When there are gaps in existing measures, the EQRO may develop new measures based on
current clinical practice guidelines or health services research. The EQRO should consider the
following questions:
•

Does the measure address accepted clinical guidelines relevant to the focus study
question?

•

Does the measure address an important aspect of care or operations that is meaningful to
MCP enrollees?

•

Do the available data sources allow the EQRO to reliably and accurately calculate the
measure? Are there any limitations on the ability to collect valid and reliable data?

•

Are all criteria used in the measure defined clearly (e.g., time periods, characteristics of
eligible enrollees, services to be assessed, and exclusion criteria)?

ACTIVITY 4: DEVELOP A PLAN TO STUDY THE POPULATION
The focus study should be designed to assess performance for all eligible enrollees in the
population being studied (e.g., a single MCP, all MCPs in the state, or all Medicaid beneficiaries
in the state). If the study focuses on a specific service (e.g., preventive dental care) or condition
(e.g., diabetes), the EQRO should include all members of the population that meet measurespecific eligibility criteria. For example, in a focus study on human papillomavirus (HPV) vaccine
rates, the EQRO should specify the population of MCP enrollees who meet the eligibility criteria
based on age or other factors (See box, Tips for Defining the Study Population).
Once the study population is defined, the EQRO should
decide whether to review performance for every enrollee in
the study population or whether performance needs to be
assessed for a representative sample of the population. The
appropriate method for the focus study may depend on
factors related to the availability and quality of data (See box,
Factors to Consider in Deciding Whether to Study the Total
Population or a Sample).
If the EQRO decides to assess performance for a sample of
the study population, it should use standard statistical

Tips for Defining the Study
Population
EQROs should carefully review
the technical specifications for
existing measures or develop
detailed specifications for new
measures to define the measureeligible population included in the
denominator for the study
variables.

More information about the Child Core Set is available at https://www.medicaid.gov/medicaid/quality-of-care/performancemeasurement/child-core-set/index.html. More information about the Adult Core Set is available at
https://www.medicaid.gov/medicaid/quality-of-care/performance-measurement/adult-core-set/index.html. More information
about measures for behavioral health clinics is available at https://www.samhsa.gov/section-223/quality-measures. More
information about the Core Quality Measures Collaborative is available at https://www.cms.gov/Medicare/Quality-InitiativesPatient-Assessment-Instruments/QualityMeasures/Core-Measures.html.
81
More information about HEDIS® is available at http://www.ncqa.org/hedis-quality-measurement. More information about
AHRQ quality measures is available at http://www.qualityindicators.ahrq.gov/.
80

286 | PROTOCOL NINE

methods to select the sample. The sample should be drawn in a way that ensures that it will
reflect the total study population. If the sample is not representative, then the focus study will not
produce valid and reliable (generalizable) results. To ensure that the sample is representative,
every enrollee in the study population should have an equal probability of being selected for the
sample and the sample should be large enough to reflect the diversity in the study population. If
the focus study results will be assessed for particular subpopulations of enrollees, the sampling
strategy should be designed so that there are sufficient numbers of enrollees in each
subpopulation included in the sample. Please refer to Appendix B for additional guidance on
sampling approaches and sample sizes.
Factors to Consider in Deciding Whether to Study the Total Population or a Sample
• If the information needed for a study variable is reliable and complete for the population in an available data source
(such as claims/encounters or enrollment data), measuring the total population might be appropriate
○ For example, if the MCP's encounter data reliably provide information about the use of prenatal care (e.g., date,
type of visit, type of provider), then the EQRO can use these records to assess performance for the total study
population
• If the EQRO has concerns about the reliability and completeness of administrative data, it may be more appropriate
to conduct the study for a sample of the population
○ For example, a study on childhood obesity that requires data on body mass index (BMI) assessments may not
be able to obtain this information in existing administrative data and may need to use medical records for a
sample of enrollees
○ Similarly, a study on prenatal and postpartum care may need to use medical records where an MCP pays for
maternity care using a bundled payment approach; in such cases, administrative data may not provide the
necessary level of detail on the number and date of prenatal and postpartum care visits

ACTIVITY 5: COLLECT DATA
Once the EQRO has established the focus study topic, question(s), variable(s), and population,
the next step is to collect data. Because data collection can be costly and burdensome, the
EQRO should first develop a data collection plan that specifies:
•

The data sources for the focus study

•

The data to be collected

•

How and when the data will be collected

•

Frequency of data collection

•

Who will collect the data

•

Instruments that will be used to collect the data

The plan should clearly specify the data sources and explain how the EQRO will ensure that the
collected data are complete and reliable for the total study population. For example, if the
EQRO will analyze claims or encounter records for an MCP, the EQRO should assess the data
to ensure that it contains consistent and complete information for all enrollees included in the
study population. Moreover, the EQRO should ensure that diagnosis and procedure codes are
used consistently across providers and that services provided in all settings are included.

PROTOCOL NINE | 287

If the EQRO plans to develop original data collection tools for the focus study, these tools
should be designed to obtain reliable results for all subpopulations included in the study. In
addition, the data collection plan should confirm that the individuals conducting the data
collection have the necessary expertise and training for the task. For example, if the focus study
requires a survey, interviewers should be trained to conduct the survey in a systematic,
unbiased manner. Similarly, if the focus study requires medical record reviews, special attention
should be given to the qualifications of the medical record reviewers, the specificity of the
guidelines for data collection, and plans for ensuring inter- and intra-rater reliability. The
reviewers should have a standard protocol for reviewing records, have the knowledge to
interpret the records, and have been trained to identify and code the information in the records
using consistent decision rules (See box, Special Considerations for Medical Record Review).
Special Considerations for Medical Record Review
• Medical record reviewers require the conceptual and organizational skills to abstract data. These skills will vary
depending on the nature of the data and the degree of professional judgment required. For example, experienced
clinical staff (such as registered nurses) should be used to extract the appropriate data from medical records to
support a judgment about whether clinical criteria are met. In contrast, trained medical assistants or medical
records clerks may collect data if the abstraction involves verifying the presence of a diagnostic test report
• Guidelines for obtaining and recording the data are essential. A glossary of terms should be developed before
data collection begins to ensure consistent interpretation among and between reviewers. In addition, reviewers
should have clear and succinct written instructions, including an overview of the study, how to complete each
section of the form, and general guidance on how to handle situations not covered by the instructions. This is
particularly important when multiple reviewers are collecting data
• Plans for ensuring inter- and intra-rater reliability are key. The number of reviewers used for a given project
affects the reliability of the data. A smaller number of staff promotes inter-rater reliability; however, it may also
increase the amount of time it takes to complete the task. The focus study should also consider and address intrarater reliability (i.e., reproducibility of judgments by the same abstractor at a different time)

ACTIVITY 6: ANALYZE AND INTERPRET STUDY RESULTS
Before beginning the focus study, the EQRO should establish a plan for analyzing and
interpreting the data. Data analysis begins with examining performance on the selected clinical
or nonclinical indicators. Accurate data analysis is essential because the state and MCPs may
implement changes in treatment and operations based on the results. The review should be
conducted using statistical analysis techniques defined in the data analysis plan.
Interpretation and analysis of the study data should involve developing hypotheses about the
causes of less-than-optimal performance and collecting data to validate the hypotheses.
Interpretation and analysis should also be based on a continuous improvement philosophy to
identify areas for improving administrative or delivery system processes.
The EQRO should conduct a quality assurance (QA) review of the analysis before it is finalized.
When reviewing the data analysis and study results, the QA reviewer should consider the
following questions:
•

Was the analysis conducted in accordance with the data analysis plan? Were conventional
methods used to conduct the analysis?

•

Are numerical results and findings presented in an accurate, clear, and easily understood
manner?

288 | PROTOCOL NINE

•

Does the analysis identify:
○

The focus study question and variables used to address the question?

○

Realistic and unambiguous targets/benchmarks for the measures?

○

Performance by key subgroups (e.g., by age, geographic location, health status, MCP)?

○

Statistical significance of differences among subgroups?

○

Factors that threaten the validity and reliability of the findings (e.g., missing data)?

Does the analysis of the study data include an interpretation of the extent to which the focus
study is successful and what follow-up activities are planned as a result?

ACTIVITY 7: REPORT RESULTS TO THE STATE
After completing the focus study (including the QA review), the EQRO will report the results to
the state in its EQR technical report (See box, Tips on Reporting on Focus Studies in EQR
Technical Reports). Because the state may use the report to meet its reporting requirements to
federal or state agencies, the state legislature, local advocacy groups, or other interested
parties, the state may need the report to include specific information presented in a specific
format. At minimum, the report should include the following information about the focus study:
•

Overall summary of findings

•

Study question and objectives

•

Methods of data collection and analysis

•

Detailed findings, including tables and graphics

•

Conclusions drawn from the data

To ensure that the report includes appropriate information in the desired format, the EQRO
should submit an outline to the state before writing up the results. The EQRO should also
confirm the audience for the report and the plans for dissemination (e.g., to CMS, MCPs,
providers, advocates, state legislators). With this information, the report can be appropriately
targeted to the intended audience.
Tips on Reporting on Focus Studies in EQR Technical Reports
• Define the study question and objectives, methods, and data sources clearly and completely. Specify the study
population and time frame
• Use tables and graphics to “tell a story” with the data; make sure to answer the study question with the data.
• If comparisons are made between subgroups, conduct tests of statistical significance to determine whether
differences are statistically meaningful
• Describe the implications of the findings for understanding current performance and for developing quality
improvement initiatives. Identify strengths and weaknesses related to quality, timeliness, and access.
• Clearly state the study limitations and caveats

END OF PROTOCOL 9

PROTOCOL NINE | 289

This page left blank for double-sided copying.

290 | PROTOCOL NINE

PROTOCOL 10. ASSIST WITH QUALITY
RATING OF MEDICAID AND CHIP MANAGED
CARE ORGANIZATIONS, PREPAID INPATIENT
HEALTH PLANS, AND PREPAID AMBULATORY
HEALTH PLANS

RESERVED.

PROTOCOL TEN

AN OPTIONAL EQR-RELATED ACTIVITY

291

This page left blank for double-sided copying.

292 | PROTOCOL TEN

APPENDIX A. INFORMATION SYSTEMS
CAPABILITIES ASSESSMENT

ACTIVITY 1: MCP COMPLETES THE ISCA TOOL
ACTIVITY 2: PERFORM PRELIMINARY ISCA REVIEW (PRE-ONSITE
ANALYSIS)
ACTIVITY 3: CONDUCT MCP ONSITE VISIT
ACTIVITY 4: COMPILE AND ANALYZE ISCA FINDINGS

BACKGROUND
This appendix defines the recommended capabilities of a managed care
plan’s (MCP’s) 82 information system (IS) to meet regulatory requirements
for managed care quality assessment and reporting, and provides an
approach the external quality review organization (EQRO) can use to
assess the strength of each MCP’s information system capabilities.
Portions of the Information Systems Capabilities Assessment (ISCA) are
voluntary; however, some components are required for the mandatory
EQR-related activities protocols. The regulations at 42 C.F.R. § 438.242
and §457.1233(d) also require the state to ensure that each MCP
maintains a health information system that collects, analyzes, integrates,
and reports data for purposes including utilization, claims, grievances and
appeals, disenrollment for reasons other than loss of Medicaid or CHIP
eligibility, rate setting, risk adjustment, quality measurement, value-based
purchasing, program integrity, and policy development.
Figure A.1 shows the interrelationship of data activities for providers,
MCPs, and EQROs. Per 42 C.F.R. § 438.242, the MCP’s information
system must be able to achieve the following:
1

Provide the state with all data elements the state deems necessary
for the mechanized claims processing and information retrieval
systems it uses for the management, monitoring, and administration
of its Medicaid or CHIP program. Collect data on enrollee and
provider characteristics as specified by the state provider and
eligibility files, and on all services received by an enrollee regardless
of payment methodology, including services sub-capitated by a MCP
to a provider, through an encounter data system or other method that
meets state requirements

APPENDIX A

ACTIVITY 5: DRAFT ISCA SUMMARY FOR EQR TECHNICAL REPORT

For the purposes of this document, the term MCP includes Managed care organizations
(MCOs), prepaid inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs),
but does not include PCCM entities.

293

Ensure that data received from providers are accurate and complete by:
○

Verifying the accuracy and timeliness of reported data

○

Screening the data for completeness, logic, and consistency

○

Collecting data from providers in standardized formats (e.g., T-MSIS) to the extent
feasible and appropriate

Make all collected data available to the state and to CMS upon request

294 | APPENDIX A

Figure A.1. Provider, MCP, and EQRO Data Activities

APPENDIX A | 295

This appendix provides an overview of the activities for assessing an MCP’s data collection,
processing, and reporting systems. The appendix concludes with information about the future of
information system assessments.
To complete this protocol, the EQRO undertakes seven activities for each MCP (Figure 9.1).

GETTING STARTED ON THE ISCA
Assessing an MCP’s information systems encompasses five consecutive activities (Figure A.2).
Figure A.2. Overview of ISCA activities

One supplemental resource is available to help EQROs conduct validation of the ISCA:
•

Worksheets for Appendix A. Information System Capabilities Assessment (ISCA) Tools,
which can be used to enable the MCP to collect standard information about its information
system, and to guide onsite information systems interviews of MCP staff

296 | APPENDIX A

ACTIVITY 1: MCP COMPLETES THE ISCA TOOL
WORKSHEET A.1

The MCP should complete the ISCA tool (Worksheet A.1) to provide standard information about
its IS and gather all requested documentation identified on a checklist at the end of the
assessment tool. The MCP should return the completed
ISCA tool and documentation to the EQRO within a
Resources for Activities 1 & 2
timeframe defined by the state.
Worksheet A.1. Information System
Capabilities Assessment (ISCA) Tool

Some states assess the capabilities of the MCP’s
information system as part of pre-contracting, contract
compliance, or contract renewal activities. The MCP
must make any previously conducted assessments
accessible to the EQRO. The EQRO should review any
such assessments as part of its ISCA review process.

• An information collection tool provided
to an MCP by the state or its EQRO to
obtain the information needed to
validate the capabilities of the MCP’s
information systems, processes, and
data, to support annual EQR-related
activities and associated EQR analysis
and recommendations

ACTIVITY 2: PERFORM PRELIMINARY
ISCA REVIEW (PRE-ONSITE ANALYSIS)

The EQRO assesses the adequacy of MCP policies and procedures based on the information
submitted by the MCP on the ISCA tool (Worksheet A.1) and its accompanying documentation.
MCP answers should be evaluated against the information system standards established by the
state to calculate and report specific plan-level performance measures, and collect and submit
encounter data to the state. The EQRO should identify sections of the ISCA that the MCP has
not fully completed. The EQRO may use the Managed Care Plan (MCP) Information System
Review Worksheet & Interview Guide (Worksheet A.2) to organize information for the site visit
interviews with MCP staff (Activity 3).

ACTIVITY 3: CONDUCT MCP ONSITE VISIT
WORKSHEET A.2

The EQRO conducts an onsite visit to the MCP to
validate the completed ISCA tool (Worksheet A.1)
and to gather additional information as needed.
The EQRO conducts interviews with MCP staff
responsible for completing the ISCA, as well as
additional staff responsible for the MCP’s
information system functions. The interviews focus
on the topics outlined in the ISCA Interview Guide
(Worksheet A.2), based on the pre-onsite analysis
of the ISCA in Activity 2. The interview with the
MCP should be closely coordinated with the MCP
onsite visit performed in Protocol 3. Assessment of
Compliance with Medicaid and CHIP Managed
Care Regulations. Refer to Protocol 3, Activity 3 for
steps in conducting a successful MCP onsite visit.

Resources for Activity 3
Worksheet A.2. Information System Review
Worksheet & Interview Guide
• A tool to conduct interviews with MCP staff
that completed the ISCA tool (Worksheet
A.1), as well as other MAP staff as needed
• These questions are intended to guide the
reviewer's discussion with MCP staff to help
validate the completed ISCA
• The questions are first organized by MCP
staff roles and then by regulatory provision

APPENDIX A | 297

ACTIVITY 4: COMPILE AND ANALYZE ISCA FINDINGS
At the conclusion of the ISCA onsite visit, the EQRO compiles and analyzes the information
gathered through the preliminary ISCA review (Activity 2) and from the onsite visit (Activity 3).
After completing its analysis, the EQRO writes a statement of findings about the MCP’s
information system. This statement should include implications of the findings for the following:
•

Completeness and accuracy of encounter data collected and submitted to the state

•

Validation and/or calculation of performance measures

•

Completeness and accuracy of tracking of grievances and appeals

•

Utility of the information system to conduct MCP quality assessment and improvement
initiatives

•

Ability of the information system to conduct MCP quality assessment and improvement
initiatives

•

Ability of the information system to oversee and manage the delivery of health care to the
MCP’s enrollees

•

Ability of the information system to generate complete, accurate, and timely T-MSIS data

•

Utility of the information system for review of provider network adequacy

•

Utility of the MCP’s information system for linking to other information sources for qualityrelated reporting (e.g., immunization registries, health information exchanges, state vital
statistics, public health data)

ACTIVITY 5: DRAFT ISCA SUMMARY FOR EQR TECHNICAL REPORT
A summary of the ISCA should be included in the EQR technical report developed by the
EQRO. This summary should include:
•

When the most recent ISCA was completed

•

The statement of the findings from the review Overall findings from the review

•

Based on findings from the ISCA, recommendations to the state relevant to EQR-related
activities and/or revisions to the state’s managed care quality strategy

THE FUTURE OF INFORMATION SYSTEMS ASSESSMENT
With increasingly sophisticated and comprehensive information systems, it is important to adapt
the way information systems’ capabilities are assessed. As information systems evolve, so will
the tools and rules with which states and EQROs assess them. As an example, information
systems may now be built on on-premise physical infrastructure, a cloud platform, or a hybrid of
both, which requires the ability to assess system security on these platforms to ensure the
privacy and security of protected health information (PHI) data. Supports for meeting existing
statutory requirements regarding privacy and security, including guidance and tools, might be
considered suitable topics for one of two optional EQR-related activities, additional performance
improvement projects (Protocol 8. Implementation of Additional Performance Improvement
Projects) or focus studies (Protocol 9. Conducting Focus Studies of Health Care quality).

298 | APPENDIX A

Given the ongoing and accelerating accumulation of health information technology (HIT)
standards, HIT certification requirements, and HIT qualifications proposed and imposed by
federal payers, organizations should anticipate changes in assessments of new information
system requirements. Two recent developments are particularly pertinent to the future of
information systems assessment:
1

In 2011, CMS began working with state agencies and other stakeholders to finalize a new
data infrastructure, the Transformed Medicaid Statistical Information System (T-MSIS). TMSIS is designed to modernize the way that states submit data about beneficiaries,
providers, fee-for-service claims, and encounters to CMS. T-MSIS (1) expands required
data elements on person-level eligibility and services; (2) captures data on providers,
managed care plans, and third-party insurance; (3) provides for improved quality of state
data; and (4) requires states to submit data monthly instead of quarterly, making the data
available sooner. CMS expects that state agencies will thoroughly audit the managed care
data to ensure that it complies with all T-MSIS requirements before submission to CMS

New efforts to implement value-based purchasing, alternative payment models, and
integrated care models will require assessment of the MCPs’ ability to track (1) bundled,
incentive, bonus, and capitated payments, (2) whether all the needed services were
delivered, and (3) how clinical quality data for performance measurement is captured and
communicated back and forth to care managers. Including all paid amounts on encounter
data provides important information to the state and CMS, enabling more data-driven
analytic methods to value-based purchasing efforts and rate development

Of continuing importance to successful information systems is the adoption and meaningful use
of certified electronic health records (EHRs). The passage in 2016 of the 21st Century Cures
Act 83—which is designed to help improve care delivery by ensuring the interoperability of health
information exchange (HIE) systems for seamless patient care through increased coordination
and continuity of health care among health care providers—highlights the significance of
certified EHRs and HIE systems as potential drivers of improvements in individual and
population health. The design and utilization of secure EHRs will become an increasingly
important element in the EQR process as is reflected in the questions included in the ISCA tool.
States and MCPs should work collaboratively in the planning and use of certified EHRs and
health information exchange systems.
States and MCPs must also coordinate their HIT planning efforts to ensure interoperability
between systems that effectively provide for future data needs to meet eligibility, enrollment,
Health Insurance Exchange, quality reporting, and delivery system reform statutory and
regulatory requirements. EQROs should continually assess MCP planning activities to ensure
alignment with and responsiveness to these initiatives. For example, this could include use of
data from bi-directional data exchange with immunization registries to support state reporting of
the CMS Core Set of Children’s Health Care Quality Measures for Medicaid and the Children’s
Health Insurance Program (CHIP) (the Child Core Set) measure on immunizations.
To learn from and share state experiences with emerging HIT and EHR initiatives that can
impact performance measure and performance improvement project outcomes reporting, CMS
strongly encourages states that contract with EQROs to include results of state HIT and EHR
See https://www.gpo.gov/fdsys/pkg/PLAW-114publ255/content-detail.html for more information about the 21st Century
Cures Act.

APPENDIX A | 299

initiatives impacted by MCP reporting in annual EQR reports. This may include successful
implementation of health information exchange with various state agencies to improve data
source collection efforts for performance measures (such as electronic clinical quality measures
or other administrative data sources used in the calculation of quality measures) or performance
improvement projects. Similarly, including lessons learned from challenging or unsuccessful HIT
initiatives are just as informative to federal and other state partners, and may also be included in
annual EQR technical reports.
END OF APPENDIX A

300 | APPENDIX A

WORKSHEETS FOR APPENDIX A
Worksheet A.1 Information System Capabilities Assessment (ISCA) Tool
Instructions. The ISCA tool is an information collection tool provided to an MCP 84 by the state or its EQRO to obtain
the information needed to validate the capabilities of the MCP’s information systems, processes, and data, with the
intent of supporting annual EQR-related activities and associated EQR analysis and recommendations as
documented in the EQR technical report. The state or its EQRO will define a timeframe in which the MCP is expected
to complete and return the tool or comparable information. For purposes of this worksheet, it is assumed the MCP will
record data in this tool. Documents from the MCP requested throughout the tool are listed in the Summary of
Requested Documentation Checklist, below.
The state and the MCP should be certain that data being reported are not only accurate today, but also have a reasonable
chance of being accurate for future reporting periods. Future accuracy can be predicted by assessing the MCP’s system
development cycle and supporting environment. Plans that lack development checkpoints and controls are much more likely
to introduce errors as systems change. The questions in this tool can be used to subjectively assess the likelihood of future
reporting anomalies. However, it should be noted that very few entities with information systems meet all the desirable
criteria. The EQRO is directed to consider the status of checkpoints and controls in its overall assessment of findings.
If the MCP’s information has been formally assessed within the last two years, please attach a copy of the
assessment report. Complete only those sections of the ISCA tool that were not covered by or have changed since
the formal assessment was conducted. If applicable, attach a copy of the MCP’s most recent information systems
analysis completed as a part of an accreditation review or third party performance measure validation process.
Note: The information requested in the ISCA pertains to the collection and processing of data for an MCP’s Medicaid
and/or CHIP line of business. In many situations, if not most, this may be no different than how an MCP collects and
processes commercial or Medicare data. However, for questions that address areas where Medicaid or CHIP data
are managed differently than commercial or other data, please provide the answers to the questions as they relate to
Medicaid or CHIP enrollees and Medicaid or CHIP data.
Any time there is a system difference between Medicaid and CHIP, it should be reported in the MCP’s responses.
However, unless noted, it is assumed that the MCP treats data from these two programs in the same manner.

MCP Contact Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, E-mail address, and date of interview, if applicable.
MCP Name:
MCP Contact Name:
Title:
Mailing address:
Phone number:
E-mail address:
Interview Date:
Type of delivery system
(check all that apply)

□ MCO □ PIHP □ PAHP
□ Other (specify): ___________________________________

Programs (please check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid and CHIP

For the purposes of the Appendix A worksheets, the term MCP includes managed care organizations (MCOs), prepaid
inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs), but does not include PCCM entities, because
42 C.F.R. Section 342, which is the basis for the requirement that states ensure maintenance of health information systems,
is only applicable to MCOs, PIHPs and PAHPs.

APPENDIX A | 301

Summary of Requested Documentation Checklist
Instructions. As you complete the ISCA tool and gather the files, please label all attached documentation as
described in the table column “Requested Document,” and when applicable by the activity number from the ISCA.
You are not limited to providing only the documentation listed below; you are encouraged to provide any additional
documentation that helps clarify an answer or eliminates the need for a lengthy response.
Check box if
document is
attached

Requested Document

Details

Previous Medicaid
Performance Measure Audit
Reports

If applicable, attach the information system analysis report
completed as a part of the MCP’s most recent accreditation
review or its most recent third party performance measure
validation process

Organizational Chart

Attach an organizational chart for your MCP. The chart should
make clear the relationship among key
Individuals/departments responsible for information
management, including performance measure reporting.

Data Integration Flow Chart

Attach a flowchart that gives an overview of the structure of
your management information system. See the example
provided in Section II-D. “Integration and Control of Data for
Performance Measure Reporting.” Be sure to show how all
claims, encounter, membership, provider, EHR, and vendor
data are integrated for performance measure reporting.

Performance Measure
Repository File Structure (if
applicable)

Attach a complete file structure, file format, and field definitions
for the performance measure repository.

Program/Query Language for
Performance Measure
Repository Reporting (if
applicable)

Attach full documentation on the software programs or codes
used to convert performance measure repository data to
performance measures.

Continuous Enrollment Source
Code

Attach a copy of the source code that you use to calculate
continuous enrollment for Medicaid or CHIP enrollees. If no
source code is use, then provide the computer program used

Medicaid Member Months
Source Code

Attach a copy of the source code/computer programs that you
use to calculate member months, member years for Medicaid
or CHIP enrollees.

Medicaid or CHIP Claims Edits

Attach a list of specific edits performed on claims as they are
adjudicated with notation of performance timing (pre- or postpayment) and whether they are manual or automated
functions.

Statistics on Medicaid or CHIP
claims/encounters and other
administrative data

Attach documentation that explains statistics reported in the
ISCA.

302 | APPENDIX A

Section 1. Background Information
1. Please select your Managed Care Model. Mark only one.
□ MCO
□ PIHP
□ PAHP
2. What year was the MCP incorporated? ______________
3. Enter your average unduplicated member enrollment for the last three years. For each column enter the reference
year.
Insurer

Year 1:

Year 2:

Year 3:

Privately insured
Medicare
Medicaid
CHIP
Other (specify)
4. Has your organization ever undergone a formal information system capability assessment?
□ Yes
□ No (GO TO SECTION 2)
4a. If yes, who performed the assessment? ______________
4b. When was the assessment completed? ______________
4c. Please provide a copy of the results of each assessment performed within the past 2 years.

APPENDIX A | 303

Section 2. Information Systems: Data Processing Procedures & Personnel
These questions attempt to determine the stability and expertise of the information system department. Responses
can provide additional insight into the development cycle responses. Outsourcing means using non-employees to get
the work done, sometimes off-site, in which case project specification, management, coordination, and acceptance
become key success factors. Enter an educated guess if the turnover rate is unknown.
1. What type of system or repository does your organization use to store Medicaid and CHIP claims and encounter
data?
2. Is this data system or repository located on-site or located in the cloud?
□ Onsite (GO TO QUESTION 3)
□ In the cloud
2a. If in the cloud, which cloud provider hosts the data?
3. How would you characterize this system or repository? Mark all that apply.
□ Relational database management system (DBMS)
□ Network
□ Hierarchical DBMS
□ Flat file
□ Indexed
□ Proprietary
□ Other: Specify _____________
□ Don’t know
4. Into what repository or DBMS(s), if any, do you extract relevant Medicaid or CHIP encounter/claim/enrollment
detail for analytic reporting purposes?
5. How would you characterize the repository/DBMS(s)? Mark all that apply.
□ Relational DBMS
□ Network
□ Hierarchical DBMS
□ Flat file
□ Indexed
□ Proprietary
□ Other: Specify _____________
□ Don’t know
6. What programming language(s) do you use to create Medicaid/CHIP data extracts or analytic reports?
___________________________________________________________________
6a. How many staff are trained and capable of modifying these programs?
_______________
7. Do you calculate defect rates for programs?
□ Yes
□ No (GO TO QUESTION 8)
7a. If yes, what methods do you use to calculate the defect rate?
7b. What was the most recent time period?

304 | APPENDIX A

7c. What were the results?
8. Approximately what percentage of your organization’s programming work is outsourced?

_______%

9. What is the average years of experience among those staff who perform programing and data analysis in your
organization?
10. Approximately how many resources (time, money, etc.) are spent on training per programmer and analysis staff
per year?
Number of hours: ___________________
Dollars spent: $__________________
Other resources (specify): ___________________
10a. What type of training for programmers is provided?
11. What is the turnover rate for your programming and analysis staff for each of the last 3 years (new staff per
year/total staff)?
Year 1 (20xx): _____ % Year 2 (20xx): _____ % Year 3 (20xx): _____ %
12. Does your organization follow a standard software development methodology (SDLMC)?
□ Yes
□ No (GO TO QUESTION 13)
12a. Outline the steps of the maintenance cycle for your state’s mandated Medicaid and CHIP reporting
requirement(s). Include any tasks related to documentation, debugging, roll out, training, etc. The level of
detail should result in 10–25 steps in the outline.
13. Does your organization use version control software for change management and deployment to the production
environment?
□ Yes
□ No (GO TO QUESTION 14)
13a. If yes, what product is used?
Note (Q13a): The information system department should follow a standardized process when
updating and revising code. This process should include safeguards that ensure that the correct
version of a program is in use

13b. Do all programmer and analysis staff and all of your systems use this product for development and
deployment?
□ Yes
□ No
14. How does your organization know if changes to the claims/encounter/enrollment tracking system affect required
reporting to the state Medicaid or CHIP program (e.g., what prompts your organization to change these systems)?
Note (Q14): A specific individual within the organization should be responsible for determining the
impact of any changes made to the plan’s claims/encounter/enrollment tracking systems. The plan
should have in place a system for triggering information system staff to update the programs.

APPENDIX A | 305

15. Who is responsible for your organization meeting the state Medicaid and CHIP reporting requirements? Mark all
that apply.
□ CEO
□ CFO
□ COO
□ CCO
□ Other (Specify) _____________________

306 | APPENDIX A

Section 3. Staffing
1. Describe the Medicaid or CHIP data processing organization in terms of staffing and the expected productivity
goals. What is the overall daily, monthly, and annual productivity of the overall department and by processor?
Note (Q1): Unusually high productivity goals can affect the accuracy and quality of a processor’s
work.

2. Describe processor training from new hire to refresher courses for seasoned processors.
Note (Q2): New hires should be provided with on-the-job training and supervision. Supervisors
should closely audit the work of new hires before concluding the training process. Seasoned
processors should have occasional refresher courses and training concerning any system
modifications.

APPENDIX A | 307

Section 4. Security
1. Does your organization have a disaster recovery (DR) plan and DR system?
□ Yes
□ No (GO TO QUESTION 6)
2. Where is the DR system located?
3. Does it provide failover capability?
4. How long does it take to switch over to the DR system when the primary system fails?
5. How often is the DR system tested?
6. How frequently are system backups performed?
7. Where are backup data stored?
8. How and how often are the backups tested to make sure that the backup procedure is functioning properly?
9. How is Medicaid or CHIP data corruption prevented due to system failure or program error?
Note (Q9): A back-up procedure will protect the data from destruction due to system failure and
program error. Plans can also institute additional safeguards to protect data from being written
over during these processes.

10. Describe the controls used to assure that all Medicaid and CHIP claims data entered into the system are fully
accounted for (e.g., batch control sheets).
Note (Q10): The plan should have a process in place that ensures that all claims/encounters that
have been logged as received are entered into the system and processed.

11. Describe the provisions in place for physical security of the computer system and manual files:
Premises:
Documents:
Computer facilities:
Desktops, laptops and mobile devices:
Note (Q11): The system should be protected from both unauthorized usage and accidental
damage. Paper based claims/encounters should be in locked storage facilities when not in use.
The computer system and terminals should be protected from unauthorized access using a
password system and security screens. Passwords should be changed frequently and should be
re-set whenever an employee terminates

12. Describe the steps taken to verify that the MCP’s information system processes for protecting PHI, including its
encryption methods, are compliant with Federal Information Processing Standards Publication (FIPS) 140-2 (for

308 | APPENDIX A

more information on the FIPS 140-2 process and validation list, please review the FIPS 140-2 related documents
at https://www.hhs.gov/sites/default/files/ocr/privacy/hipaa/administrative/securityrule/fips1402.pdf?language=es).
12a. Provide the results of the most recent FIPS 140-2 tests completed on the MCP’s information system.
13. Describe the procedures in place to determine which system users may access levels of the system that include
PII. Please identify the job titles and responsibilities of each system user with access to systems that include PII.
14. Describe the methods in place to allow those with access to PII to only access the minimum amount of
information necessary to perform their job.
15. Identify training and awareness provided to personnel (system owners, managers, operators, contractors and/or
program managers) using the system to make them aware of their responsibilities for protecting the information
being collected and maintained.
16. Describe the process and guidelines in place with regard to the retention and destruction of PII.
17. Describe, briefly but with specificity, how the PII will be secured in the system using administrative, technical, and
physical controls.
18. If you employ cloud-based technology, describe the provisions in place to secure the virtual system.
19. If you utilize remote network access to connect users with the MCP’s secure networks via the internet, describe
the provisions in place to secure the network against unauthorized access.
20. Which staff position(s) is responsible for the security and user administration task that grants access to the
system?
21. Which staff positions have access to what levels of the system?
22. Can your programming and analysis staff access the production system or only the development system?
□ Production system only
□ Development system only
23. How often must passwords be changed?
24. How quickly are logons deactivated after employee terminations and resignations?
25. Describe your patch management protocols and processes.
26. What other individuals have access to the computer system? Customers? Providers? Describe their access and
the security that is maintained restricting or controlling such access.
Note (Q26): Both members and providers should have their access limited to read-only so that
they cannot alter any files. They should be given access to only those files containing their own
patients or members. Customers should be prevented from accessing highly confidential patient
information by being given “blinded” patient names and “scrambled” ID numbers, or restricted
access to particular files.

APPENDIX A | 309

Section 5. Data Acquisition Capabilities
The purpose of this section is to obtain a high-level understanding of how you collect and maintain administrative
data (claims and encounter data), enrollment information, data on ancillary services such as prescription drugs.
A. Administrative Data (Claims and Encounter Data)
These questions request information on input data sources (e.g., electronic claims and paper) and on the transaction
system(s) you use.
1. How are data submitted (e.g. electronically, on paper or both)?
□ Submitted electronically
□ Submitted on paper
□ Submitted both electronically and on paper
1a. What percent of data are submitted electronically?
1b. What formats are used?
1c. Is there a front-end web portal available for data submissions?
□ Yes
□ No
2. Do you use standard claims or encounter forms for the following? Mark yes or no for each data source. If yes,
please specify (e.g., CMS1500, UB 94).
Data source

Yes

If “Yes,” please specify

Hospital
Physician
Drug
Nursing home
Home health
Mental health
Dental

Note (Q2): Plans that do not use either CMS 1500 or UB 92 forms may be using forms they
developed themselves. If a plan is using its own forms, these forms should be reviewed to ensure
they are capturing the following key data elements: patient identification information (Medicaid
ID, name, date of birth, gender), provider identifying information (national provider identifier (NPI),
Tax ID, name), date of service, place of service and diagnoses and procedure codes. An
evaluation of their forms to ascertain adequacy and completeness of data collection may be
necessary.

3. We would like to understand how claims or encounters are submitted to your plan. We are also interested in an
estimate on an annual basis of what percentage (if any) of services provided to your enrollees by all providers
serving your Medicaid and CHIP enrollees are NOT submitted as claims or encounters, and therefore, are not
represented in your administrative data. Please fill in the following table with the appropriate percentages:

310 | APPENDIX A

Claims or Encounter Types
Medium

Hospital

PCP

Specialist
Physician

Dental

Mental health/
substance abuse

Drug

Other

Claims/encounters
submitted
electronically
Claims/encounters
submitted on paper
Services not
submitted as claims or
encounters
TOTAL

100%

Note (Q3): Since paper forms need to be entered into a plan’s system, processing paper forms is
prone to error. If a plan is receiving more that 50 percent of its data on paper forms, verify the
data checks the plan uses to test processor accuracy. Electronic data submission should also
undergo data edits and validity checks. Plans with a high percentage of unavailable data for a
particular category will have difficulty reporting measures that use that category. For example, a
plan receiving no drug data from its vendor would not be able to report the HEDIS® measures for
Outpatient Drug Utilization.

3a. For each type of claims or encounter type for which some percentage are not represented in your
administrative data, please explain why such activity is not reported.
4. In the following table, please enter an “R” in the appropriate cell if the following data elements (data fields) are
required by you for providers, for each of the types of Medicaid claims/encounters identified below. Note that each
of these elements is required by T-MSIS, and that the MCP’s data elements should align with T-MSIS
requirements:
Claims or Encounter Types
Medium
Patient

Hospital

PCP

Specialist
Physician

Dental

Mental health/
substance abuse

Drug

Other

gendera

Patient date of birth
and age
ICD9/10 Diagnosis
Codes
Procedure Code
Types:
CPT-4/HCPCS
National Drug
Code (NDC)
Universal Product
Code (UPC
Manufacturer Part
Number (MPN)
First date of service
Last date of service
Quantity of Service

APPENDIX A | 311

Claims or Encounter Types
Medium

Hospital

PCP

Specialist
Physician

Dental

Mental health/
substance abuse

100%

Drug

Other

Revenue code
Provider NPI
Provider specialty
TOTAL

100%

a
UN, M, or F (UN = the gender of a person could not be uniquely defined as male or female; M = Male; F = Female). Please see
AHRQ’s Administrative Gender Value Set document at https://ushik.ahrq.gov/ViewItemDetails?&system=mu&itemKey=86667000
for more information.

Note (Q4): Standard measures of plan performance such as Medicaid HEDIS® are dependent upon
the availability of the fields listed above. If procedure codes or diagnosis codes are not available,
the data will not include the necessary level of detail to report performance measures.

5. In the following table, please enter how many diagnoses and procedures are captured on each claim and on each
encounter:
Claim
Diagnoses

Encounter
Procedures

Diagnoses

Procedures

Institutional data
Provider/Provider group data

Note (Q5): All diagnosis codes types should be standard, nationally recognized codes, rather than
plan-specific codes. Diagnosis code fields should include all diagnosis codes needed to identify
the reason for the encounter, and all relevant comorbidities and complications should be
included. Each service rendered or product dispensed should be identified with the appropriate
identifier.

6. Can you distinguish between principal and secondary diagnoses?
□ Yes
□ No
Note (Q6): Some plans will consider the first diagnosis on the claim to be principal. Other plans
determine the principal diagnosis by selecting the most expensive condition represented.

6a. If “Yes” to 6, above, how do you distinguish between principal and secondary diagnoses?
7. Please explain what happens if a Medicaid or CHIP claim or encounter is submitted and one or more required
fields are missing, incomplete, or invalid. For example, if diagnosis is not coded, is the claims examiner required
by the system to use an on-line software product like AutoCoder to determine the correct ICD-10 code?
Institutional Data:
Professional Data:
8. How is the MCP able to distinguish backend-system-assigned data versus data submitted by the service
provider?

312 | APPENDIX A

9. What steps do you take to verify the accuracy of submitted information (e.g., procedure code, diagnosis edits,
gender-diagnosis edits, gender-procedure code edits)?
Institutional Data:
Professional Data:

Note (Q9): Plans will often verify that the information in procedure code and diagnosis code fields
are valid codes. Plans may also verify that diagnosis and procedure codes are appropriate for age
and gender. For example, a claim with a procedure of hysterectomy should be for a female
patient.

10. Under what circumstances can claims processors change Medicaid or CHIP claims/encounter information?
Note (Q10): If processors are given the ability to modify claims/encounter information, the
accuracy of that information could be affected either negatively or positively. Processors may
simply correct data that was submitted incorrectly, which would increase the quality of the data.
However, processors may also change diagnosis and procedure codes which could result in a loss
of coding specificity. Does the plan check processed data against paper claims?

11. Identify any instance where the content of a field is intentionally different from the description or intended use of
the field. For example, if the dependent’s SSN is unknown, do you enter the member’s SSN instead?
Note (Q11): Changing the content of a field can create data processing issues. For example, if the
enrollee’s SSN is used as an ID for a number of dependents, the claim may be given the age and
sex of the member rather than the actual patient. The use of the enrollee’s SSN would make it
difficult to track the dependent’s experience over time.

12. How are Medicaid or CHIP claims/encounters received from each of the following sources? Please mark one
column per source:
Source

Received directly from provider

Received through an intermediary

Hospital
Physician
Pharmacy
Nursing home
Home health
Mental health
Dental
Other
12a. If the data are received through an intermediary, what changes, if any, are made to the data? Please answer
for each source received through an intermediary in the table above.
Note (Q12): Intermediaries that are processing the data, such as a pharmacy benefit firm, could
modify the data, creating a data set that is inconsistent with the plan’s data. The intermediary
may define field content differently or may not be using the same fields as the plan, making it
difficult to integrate the intermediary’s data into the plan’s systems. All data submitted through an
intermediary should be monitored for quality by the plan.

APPENDIX A | 313

13. In the following table, please estimate the percentage of Medicaid or CHIP claims/encounters that are coded
using the following coding schemes:
Coding
scheme

Inpatient
diagnosis

Inpatient
procedure

Ambulatory/outp
atient diagnosis

Ambulatory/outpatient
procedure

Drug

ICD-10 CM
CPT®-4
HCPCS
DSM-IV
National Drug
Code
Internally
developed
Other: Specify
Not required
TOTAL (can be
greater than
100% if a
claims type is
subject to more
than one
coding system)

Note (Q13): If a plan is using internally-developed coding schemes, the state should verify
whether this coding can be mapped to standard coding such as ICD-10 or CPT-4. If the coding can
be translated for reporting purposes (Medicaid HEDIS® requires diagnosis and procedure codes),
the plan should provide information on the level of specificity with which the coding maps to
standard coding (e.g., three-digit specificity or five-digit specificity). If the mapping has a low
level of specificity, information on co-morbidities and complications may not be retained during
translation.

14. Please list all information systems through which service and utilization data for the Medicaid or CHIP population
is processed.
Note (Q15): Each upgrade or consolidation of the plan’s information system has the potential to
damage the quality of the data. For example, data could be lost or corrupted during a system
conversion, or a new system could limit a plan’s access to historical data. Changes in data quality
and access will affect the plan’s ability to report performance measures and utilization. The plan
should have a fallback option, such as parallel operations.

15. Please describe any major systems changes or updates that have taken place in the last three years in your
Medicaid or CHIP claims or encounter system (be sure to provide specific dates on which changes were
implemented). Check all that apply
□ New system installed to replace old system
□ New system purchased and installed to replace most of old system; old system still used
- Major enhancements to old system
□ If enhancements were made to the old system, please summarize what enhancements were made and
whether (and if so, how) the enhancements have impacted historical data:

314 | APPENDIX A

□ New product line adjudicated on old system
□ Conversion of a product line from one system to another
Note (Q15): When a plan undertakes any major system changes such as conversion to a new
system, the system changes could affect data quality. Data quality problems include corruption of
data, loss of data, and loss of the level of detail within the data. The implementation of a new
system can also affect the accessibility of historical data.

16. How many years of Medicaid or CHIP data are retained on-line?
16a. How is historical Medicaid or CHIP data accessed when needed?
Note (Q16a): Due to system constraints, a plan may remove historical data and place it in off-line
storage. The MCP’s ability to report on experience spanning several years of data could be
affected by the accessibility of the data stored off-line.

17. What percent of your Medicaid or CHIP data is processed on-line vs. batch? If batch, how often are batch jobs
run?
18. Describe your policy regarding Medicaid or CHIP claim/encounter audits.
18a. Are Medicaid or CHIP encounters audited regularly or randomly?
□ Regularly
□ Randomly
18b. What are the standards regarding timeliness of processing?
Note (Q18b): Plans should be performing random periodic audits of their encounter data to
determine the quality of data processing. Plans that do not perform audits at least annually are
not closely monitoring the quality of data processing. Plan standards regarding timeliness of
processing will influence the lag time for encounter data processing.

19. Please describe system edits that are targeted to field content and consistency. Are diagnostic and procedure
codes edited for validity?
Note (Q19): MCPs should have an established, standard set of edits that verify field content and
consistency. For example, a field content data edit would verify that a valid date is entered into
the date of service field. Key fields which should be edited include patient identifying information
(Medicaid ID, name, date of birth, sex), provider identifying information (name, tax ID, type), date
and place of service, and diagnosis and procedure codes. The quality of diagnosis and procedure
coding will affect the validity of reports and performance measures submitted by the MCP/PIHP.

20. Please complete the following table for Medicaid and CHIP claims and encounter data and other Medicaid and
CHIP administrative data. Attach any documentation that should be reviewed to explain the data that is being
submitted.
Item

Claims

Encounters

Other administrative data

Percent of total service volume
Percent complete

APPENDIX A | 315

Item

Claims

Encounters

Other administrative data

How are the above statistics
quantified?
Incentives for data submission

Note (Q20): MCPs with claims data comprising more than 50 percent of their total service volume
are likely to have a more complete representation of total MCP experience than MCPs that rely
heavily on encounter data. While providers have an incentive to submit claims in order to receive
payment for services, they do not always have incentives to submit encounter information. If an
MCP does not offer providers an incentive, or does not require the submission of encounter data,
the MCP may not receive data for every encounter. Other administrative data collected by an MCP
could include data from pharmacy or laboratory vendors.

21. Describe the Medicaid or CHIP claims/encounter suspend (“pend”) process including timeliness of reconciling
pended services. What percentage of claims are suspended or pended?
Note (Q21): Pended claims/encounters are those claims/encounters that have been suspended
during processing because they failed data quality edits or violated provider payment parameters.
Information on these claims and encounters will not be available for reporting until they have
been reconciled and processed into the system.

22. Describe how Medicaid or CHIP claims are suspended/pended for medical review, for non-approval due to
missing authorization code(s) or for other reasons. What triggers a processor to follow up on “pended” claims?
How frequent are these triggers?
Note (Q22): Review and processing should not be handled by the same employee. A system should
be in place which encourages the processor to follow-up on the status of claims in review that
have not yet been approved to ensure they are resolved.

23. Are any of your Medicaid or CHIP services/providers capitated?
23a. If yes, have you conducted studies on the completeness of the information collected on capitated services?
23b. If yes, what were the results?
Note (Q23b): Because provider payment for capitated services is not determined by the encounter
data submitted, providers do not have an incentive to submit complete and accurate information
on every service provided. Data on capitated services often does not include the same level of
detail as fee-for-service claims information. Per service pricing information may not be available
when providers are paid on a capitated basis but at least the amount of the capitation payment
should be available. Plans should be aware that capitated data is less complete and should audit
the data at least annually to monitor its quality.

24. In the following table, enter the claim/encounter system(s) for each product line offered to Medicaid or CHIP
enrollees.
Note (Q24): Typically, there is just one product line offered to Medicaid or CHIP enrollees, but
there may be some circumstances in which an MCP offers additional product lines to the state
(e.g., partial risk products, premium assistance programs).

316 | APPENDIX A

Medicaid
Systems Used to Process
____________________

Product Line:
____________________

Fee-for-service (indemnity)
claims
Capitated service encounters
Clinic patient registrations
Pharmacy claims
Other (describe)

CHIP (if applicable)
Systems Used to Process
____________________
Fee-for-service (indemnity)
claims
Capitated service encounters
Clinic patient registrations
Pharmacy claims
Other (describe)
25. Beginning with receipt of a Medicaid or CHIP claim in-house, describe the claim handling, logging, and processes
that precede adjudication. Describe the following: When are claims assigned a document control number and
logged or scanned into the system? When are claims stored using document imaging? If there is a delay in
document imaging, how do processors access a claim that is logged into the system, but is not yet filmed?
25a. Please describe each system or process that is involved in adjudicating:
-

A professional encounter(s) for a capitated service (e.g., child immunizations that arrive separately from
the office visit)

A hospital claim for a delivery or for a newborn that exceeds its mother’s stay

Note (Q25a): Professional encounters arriving separately from an office visit may not be
processed as quickly as the actual office visits. If these encounters are treated as “non-standard”
events, the plan may not be able to easily link these encounters with the related office visit. For
example, newborns exceeding a mother’s stay may have their hospital stay split into two parts.
The part of the stay which coincides with the mother’s hospitalization may be processed on the
mother’s claim and the remainder of the stay could be processed separately. Processing the
newborn’s stay as two separate claims could affect the plan’s ability to report accurately on
newborn hospital utilization.

25b. Discuss which decisions in processing a Medicaid or CHIP claim/encounter are automated, which are
prompted by automated messages appearing on the screen, and which are manual. Document the
opportunities a processor has for overriding the system manually. Is there a report documenting overrides or
“exceptions” generated on each processor and reviewed by the claim supervisor? If so, please describe this
report.
25c. Are any outside parties or contractors used to complete adjudication, including but not limited to:

APPENDIX A | 317

Bill auditors (hospital claims, claims over a certain dollar amount)
□ Yes
□ No

Peer or medical reviewers
□ Yes
□ No

Sources for additional charge data (usual & customary)
□ Yes
□ No

Bill “re-pricing” for carved out benefits (mental health, substance abuse)
□ Yes
□ No

Other
□ Yes (If yes, please provide additional information)
□ No

25d. How are these data incorporated into your organization’s data?
Note (Q25d): If outside parties are used, the plan should be incorporating data generated by those
parties into the system. The data should first be run through the plan’s data quality checks to
verify its accuracy and completeness.

25e. Describe the system’s editing capabilities that assure that Medicaid and CHIP claims are correctly
adjudicated
-

Attach a list of the specific edits that are performed on claims as they are adjudicated, and note (1)
whether the edits are performed pre- or post-payment, and (2) which are manual functions and which are
automated functions.

Note (Q25e): When reviewing plan adjudication edits, the state should concentrate on edits which
affect the data fields that are used to generate plan performance measures and reports. Are
outliers for length of stay and charges edited? Utilizing an automated editing process provides
more consistent results that do not require processor judgment. Edits that are performed prepayment can prevent invalid data from being incorporated into the system.

25f. Discuss the routine and non-routine (ad hoc or special) audits that are performed on claims/encounters to
assure the quality and accuracy and timeliness of processing. In your response, note which audits are
performed per processor, which rely on targeted samples, and which use random sampling techniques. What
is the total percentage of claims on-hand that are audited through these QA processes? How frequently do
these audits occur?
Note (Q25f): This item is not relevant in instances where the EQRO is performing encounter data
validation. When reviewing edits that are used to determine processor accuracy, consider that
these edits will not provide information on the quality of the initial provider data submission. The
audit plan should include random sampling techniques to provide an overall picture of quality.
Plans will often concentrate on auditing complicated or aberrant claims/encounters rather than
using a random sample. The plan should have instituted a process for sharing audit results with
the processor to facilitate quality improvement.

318 | APPENDIX A

25g. Please describe how Medicaid and CHIP eligibility files are updated, how frequently and who has “change”
authority. How and when does Medicaid and CHIP eligibility verification take place?
25h. How are encounters for capitated services handled by payment functions? What message appears to notify
processors that they are handling a capitated service?
25i. Describe how your systems and procedures handle validation and payment of Medicaid claims when
procedure codes are not provided.
Note (Q25i): Plans requiring valid procedure coding for all claims/encounters will have more
detailed data available for reporting and analysis. However, these plans may allow processors to
supply missing codes using a code book or override the system using an unspecified code. A
number of plans use programs such as the GMIS AutoCoder product to fill in missing codes. When
a plan supplies missing codes, the coding can be less accurate than codes supplied directly by
the provider of service.

26. Describe all performance monitoring standards for Medicaid and CHIP claims/encounters processing. Provide the
results of a recent performance monitoring activity.
26a. How is performance against targets figured into the official performance appraisal process? Into processor
and supervisor compensation?
B. Enrollment System
1. Please describe any major changes/updates that have taken place in the last three years in your Medicaid or
CHIP enrollment data system. Include the specific dates on which changes were implemented. For example:
-

New enrollment system purchased and installed to replace old system

New enrollment system purchased and installed to replace most of old system; is the old system still used?

Major enhancements to old system; what kinds of enhancements, and what impact on your historical data?

New product line members stored on old system
Note (Q1): Changes to a plan’s enrollment system requiring data conversion and data integration
can create data quality problems. Implementing a new enrollment system could lead to a loss of
access to data on the old system, or the assignment of new member numbers for all enrollees.
Data conversion and integration can also limit a plan’s ability to track an enrollee’s enrollment
history. When a new product line is added to an existing system, a plan may need to make the
new data fit the older process, therefore modifying the system to “handle” new information.
Implementing such modifications can be difficult for a plan that has been using the same system
for a number of years. The level of enrollment detail retained can be affected by such
modifications.

2. In your opinion, have any of these changes influenced, even temporarily, the quality and/or completeness of the
Medicaid or CHIP data that are collected? If so, how and when?
Note (Q2): Consider whether changes in data quality will affect the validity of the data submitted
to the state.

3. How does your plan uniquely identify enrollees?

APPENDIX A | 319

Note (Q3): Major changes to an MCP’s enrollment system could involve the conversion of
membership data to a new system. When MCPs convert members, they may change the enrollee’s
ID number, making it difficult to track the enrollee’s enrollment pattern across time. Changes to
the enrollment system could also lead to a loss of data for specific patients.

4. How do you handle enrollee disenrollment and re-enrollment in the Medicaid or CHIP product line? Does the
member retain the same ID?
Note (Q4): Enrollees should have a single ID number to facilitate tracking their experience.
However, some plans change an enrollee’s ID number when the enrollee re-enrolls. Experience for
enrollees who have switched ID numbers will be more difficult to track. Dependents using an
enrollee’s ID are also difficult to identify for reporting purposes. For example, children without a
unique ID could affect the ability of the plan to report on low birth-weight babies, childhood
immunizations, and asthma inpatient admissions. This is an important point. EQROs should give
higher “grades” to plans that use strong methods of identifying enrollees.

5. Can your systems track enrollees who switch from one product line (e.g., Medicaid, commercial plan, Medicare)
to another?
□ Yes
□ No
5a. Can you track an enrollee’s initial enrollment date with your MCP?
□ Yes (GO TO QUESTION 5C)
□ No
5b. If not, is a new enrollment date assigned when a member enrolls in a new product line?
□ Yes
□ No
5c. Can you track and link previous claim/encounter data across product lines?
□ Yes
□ No
6. Under what circumstances, if any, can a Medicaid or CHIP member exist under more than one identification
number within your MCP’s information management systems? Under what circumstances, if any, can a member’s
identification number change?
7. How does your MCP enroll and track newborns born to an existing Medicaid or CHIP enrollee?
7a. If your MCP has a Medicare product line, describe how your enrollment systems link individuals
simultaneously enrolled in both your Medicare product line and the Medicaid plan product line.
8. Is claim/encounter data linked for Medicare/Medicaid dual eligibles so that all encounter data can be identified for
the purposes of performance measure reporting?
□ Yes
□ No
8a. Is claim/encounter data linked for individuals enrolled in both a Medicare and Medicaid plan so that all
encounter data can be identified for the purposes of performance measure reporting?
□ Yes
□ No

320 | APPENDIX A

9. How often is Medicaid and CHIP enrollment information updated?
Note (Q9): Enrollment information should be updated real-time, daily, or weekly.

10. How is Medicaid and CHIP continuous enrollment being defined? In particular, does your system have any
limitations that preclude you from fully implementing continuous enrollment requirements exactly as specified in
the state performance measure requirements?
11. Please attach a copy of the source code that you use to calculate Medicaid/ CHIP continuous enrollment.
12. How do you handle breaks in Medicaid or CHIP enrollment, e.g., situations where a Medicaid enrollee is
disenrolled one day and re-enrolled the next simply for administrative reasons? Does this affect your continuous
enrollment calculations?
13. Do you have restrictions on when Medicaid or CHIP enrollees can enroll or disenroll? Please describe.
14. How do you identify and count the following:
Medicaid member months?
Medicaid member years?
15. Please list all data from which claims/encounters for the Medicaid or CHIP product line are verified.
Note (Q15): Eligibility of the patient should be verified before claims and encounters are
processed. Dates of enrollment and disenrollment are key reporting fields for Medicaid HEDIS®
measures. Eligibility status is dynamic for Medicaid beneficiaries and should be updated
frequently. Eligibility status should also be verified before data is submitted to the state.

16. Does the plan offer vision or pharmacy benefits to its Medicaid or CHIP members that are different from the vision
or pharmacy benefits offered to its commercial enrollees (within a given contract or market area)?
□ Yes
□ No (GO TO SECTION C, ANCILLARY SYSTEMS)
16a. If vision benefits vary by benefit package, outline the different options available. How are enrollees
tracked?
16b. If pharmacy benefits vary by benefit package, outline the different options available. How are enrollees
tracked?
C. Ancillary Systems
Use this section to record information on stand-alone systems or benefits provided through subcontracts, such as
pharmacy or mental health/substance abuse.
1. Does your MCP incorporate data from one or more third-parties to calculate any of the following Medicaid and
CHIP quality measures? If so, which measures require third-party data?

APPENDIX A | 321

Note (Q1): The measures listed in the following table are examples of measures that can be
calculated with administrative data and align with CMS quality measurement initiatives as of
2017. The state and EQRO should tailor this table to list those measures that the state requires its
MCP contractors to produce and any other measures in which the state is interested.

Measure

Third-Party Data Source

Childhood Immunization Status (IMA-CH)
Immunizations for Adolescents (IMA-CH)
Well-Child Visits in the Third, Fourth, Fifth and Sixth Years of Life
(W34-CH)
Adolescent Well-Care Visit (AWC-CH)
Prenatal and Postpartum Care: Timeliness of Prenatal Care (PPCCH)
Frequency of Ongoing Prenatal Care (FPC-CH)
Developmental Screening in the First Three Years of Life (DEVCH)
Chlamydia Screening in Women Ages 16–20 (CHL-CH)
Child and Adolescent Access to Primary Care Practitioners (CAPCH)
Percentage of Eligibles Who Received Preventive Dental Services
(PDENT-CH)
Weight Assessment and Counseling for Nutrition and Physical
Activity for Children/Adolescents (WCC-CH)
Follow-Up Care for Children Prescribed AttentionDeficit/Hyperactivity Disorder (ADHD) Medication (ADD-CH)
Follow-Up After Hospitalization for Mental Illness: Ages 6–20
(FUH-CH)
Ambulatory Care: Emergency Department (ED) Visits (AMB-CH)
Cervical Cancer Screening (CCS-AD)
Breast Cancer Screening (BCS-AD)
2. Describe any concerns you may have about the quality or completeness of any third-party data.
Note (Q2): If a plan is using third-party data, the plan should have a formal process in place to
validate that data before incorporating it into their information system. The plan needs to check
the third-party data for reliability, completeness and timeliness of submission.

3. Please list subcontracted Medicaid or CHIP benefits that are adjudicated through a separate system that belongs
to a third-party.
Note (Q3): Many plans contract out services for pharmacy benefits management, mental
health/substance abuse, laboratory and radiology services. If the data are processed on the thirdparty’s system, it may not be forwarded to the plan in a complete form or on a timely basis. Such
entities may also use a different method of processing resulting in data that will not merge with
or complement plan data.

322 | APPENDIX A

4. Describe the kinds of information sources available to the MCP from the vendor (e.g., monthly hard copy reports,
full claims data).
5. Do you evaluate the quality of this information?
□ Yes
□ No (GO TO QUESTION 6a)
5a. If yes, how?
Note (Q5a): All of the third-party information should be verified for accuracy before a plan loads it
into their information system. The plan and the third-party data source may not define variables
consistently or use the same reporting format.

6. Did you incorporate these vendor data into the creation of Medicaid or CHIP-related studies?
□ Yes (GO TO SECTION D)
□ No
6a. If no, why?
D. Additional Data Sources that Support Quality Reporting
This section requests any data sources beyond third party collection of claim/encounter data that support quality
reporting.
1. Does the MCP use any other data sources beyond claim/encounter data (such as, beneficiary provided data, HIE,
registry data source, vital statistics, etc.)?
□ Yes
□ No
If yes, please list additional data sources: _______________________________________________________
If yes, please describe how the MCP verifies the accuracy of the data and data exchange process for each data
source listed above.
E. Integration and Control of Data for Performance Measure Reporting
This section requests information on how your MCP integrates Medicaid and CHIP claims, encounter, membership,
provider, third-party, and other data to calculate performance rates. All questions relate to your current systems and
processes, unless indicated otherwise.
1. Please attach a flowchart outlining the structure of your management information systems, indicating data
integration (i.e., claims files, encounter files, etc.) at the most granular level you have it.
2. In consolidating data for Medicaid and CHIP performance measurement, how are the data sets for each measure
collected:
-

By querying the processing system online?

By using extract files created for analytical purposes? If so, how frequently are the files updated? How do they
account for claim and encounter submission and processing lags? How is the file creation process checked
for accuracy?

APPENDIX A | 323

By using a separate relational database or data warehouse (i.e., a performance measure repository)? If so, is
this the same system from which all other reporting is produced?

3. Describe the procedure for consolidating Medicaid or CHIP claims/encounter, member, and provider data for
performance measure reporting (whether it is into a relational database or file extracts on a measure-by-measure
basis).
3a. How many different sources of data are merged together to create reports?
3b. What control processes are in place to ensure that data merges are accurate and complete?
3c. What control processes are in place to ensure that no extraneous data are captured (e.g., lack of specificity in
patient identifiers may lead to inclusion of non-eligible members or to double counting)?
4. Describe both the files accessed to create Medicaid or CHIP performance measures and the fields from those
files used for linking or analysis. Use either a schematic or text to respond.
5. Are any algorithms used to check the reasonableness of data integrated to report Medicaid or CHIP performance
measures?
6. Are Medicaid or CHIP reports created from a third-party software product?
□ Yes
□ No (GO TO QUESTION 7)
6a. If yes, how frequently are the files updated? How are reports checked for accuracy?
7. Are the data files used to report Medicaid or CHIP performance measures archived and labeled with the
performance period in question?
□ Yes
□ No
8. Information on several types of external encounter sources is requested. In the following table, please indicate the
following for each type of delegated service:
-

Column 2. Indicate the number of third-parties contracted (or subcontracted) to provide the Medicaid or CHIP
service. Count the entities that offer all or some of the portion of the service indicated.

Column 3. Indicate whether your MCP receives member-level data for any Medicaid or CHIP performance
measure reporting from the vendor(s). Only answer “Yes” if all data received from contracted third-parties(s)
are at the member level. If any encounter-related data is received in aggregate form, you should answer “No”.
If type of service is not a covered benefit, indicate “N/A”.

Column 4. Indicate whether all data needed for Medicaid or CHIP performance measure reporting are
integrated, at the member-level, with MCP administrative data.

Columns 5 and 6. Rank the completeness and quality of the Medicaid or CHIP data provided by the third
party(s). Consider data received from all sources when using the following data quality grades:
A. Data are complete or of high quality
B. Data are generally complete or of good quality
C. Data are incomplete or of poor quality

324 | APPENDIX A

Column 7. Describe any concerns you have in ensuring completeness and quality of Medicaid or CHIP data
received from contracted third-parties. If the measure is not being calculated because there are no eligible
members, please indicate “N/A”.

Medicaid or CHIP Claim/Encounter Data from Third Parties

Type of
delegated
service

Number of
contracted
third-parties

Always receive
member-level
data from all
third party(s)
(Y or N)

Integrate thirdparty data with
MCP
administrative
data? (Y or N)

Data
completeness
(A, B, or C)

Data
quality
(A, B,
or C)

Describe
rating
concerns
with data
collection

Behavioral
health
Family
planning
Home health
care
Hospital
Laboratory
Pharmacy
Primary care
Radiology
Specialty care
Vision care
Dental for
children
9. Does your MCP use a performance measure repository?
□ Yes
□ No (GO TO QUESTION 10)
9a. If your MCP uses a performance measure repository for Medicaid or CHIP performance measures, review
the repository structure. Does it contain all the key information necessary for Medicaid or CHIP
performance measure reporting?
10. Please describe your Medicaid or CHIP report production logs and run controls.
10a. Please describe your Medicaid or CHIP performance measure report generation process.
11. How are Medicaid or CHIP report generation programs documented?
12. How does your MCP test the process used to create Medicaid and CHIP performance measure reports?
13. Are Medicaid and CHIP performance measure reporting programs reviewed by supervisory staff?
□ Yes
□ No
14. The purpose of these questions is to evaluate the Medicaid and CHIP provider compensation structure and
reporting of certain types of compensation, as this may influence the quality and completeness of data. Please
identify the percentage of member months in your plan contributed by Medicaid members whose primary care
providers and specialists are compensated through each of the following payment mechanisms:

APPENDIX A | 325

Payment mechanism

Primary care physician

Specialist physician

100%

Salaried
Fee-for-Service, no withhold or bonus
Fee-for-Service, with withhold
Please specify % withhold:___________
Fee-for-Service with bonus
Bonus range: _____________________
Capitated - no withhold or bonus
Capitated with withhold
Please specify % withhold:___________
Capitated with bonus
Bonus range: _____________________
Global/bundled payments
Other: (Specify)___________________
TOTAL

Note (Q14): Timeliness and completeness of provider data submissions often varies by
contracting arrangement. Salaried providers work directly for the MCP and will submit data on a
timely basis if data submission is a parameter in their contract with the MCP. Fee-for-service
providers have the largest incentive to submit accurate and complete data since their payment
depends upon it. Capitated providers will need incentives to submit accurate and complete data.
Their compensation should be linked to data submission, which can be done through the use of
bonuses and withholds. For example, lag times may differ by compensation arrangement as
follows: Capitation/Salaried-no lag, Fee-for-Service - 60 day lag, Hospital - 45 day lag

15. How are bonuses and penalties captured within your system? Is this information part of your standardized
reporting?
15a. Is the underlying data that determines whether and the extent of bonuses and penalties captured in your
system? Is this information part of your standard reporting?
15b. For bundled/global payments, how does your system capture information about the individual services
provided for this bundled/global payment? Is this information part of your standardized reporting?
15c. Does your system capture clinical data for quality measurement purposes for providers who receive
bundled/global payments? Is this information part of your standardized reporting?
16. Please describe how Medicaid or provider directories are updated, how frequently, and who has “change”
authority.
16a. Does your MCP maintain provider profiles on its website?
□ Yes
□ No (GO TO QUESTION 17)
16b. If yes to “16a,” what provider information is maintained in on the website (e.g., languages spoken, special
accessibility for individuals with special health care needs). Other? Please describe:

326 | APPENDIX A

17. Does your MCP maintain provider profiles on its information system?
□ Yes
□ No (GO TO QUESTION 18)
Note (Q17): Provider directories should be updated to reflect changes in provider status to
prevent members from selecting providers no longer under contract with the plan. The plan should
have adequate security procedures in place to restrict the number of individuals who can access
confidential provider information and institute changes in status.

17a. If yes to “17,” what provider information is maintained in the provider profile database (e.g., languages
spoken, special accessibility for individuals with special health care needs). Other? Please describe.
18. How are Medicaid or CHIP fee schedules and provider compensation rules maintained? Who has updating
authority?
Note (Q18): Since providers consider fee schedule and compensation information to be
confidential, access to this information should be restricted by the MCP. The MCP should have
standardized process for updating and maintaining this information.

19. Are Medicaid or CHIP fee schedules and contractual payment terms automated? Is payment against the
schedules automated for all types of participating providers?
Note (Q19): Manual payment processes are more prone to error and reduce processing speed.

END OF WORKSHEET A.1

APPENDIX A | 327

Worksheet A.2 Information System Review Worksheet & Interview Guide
Instructions. EQROs can use this managed care plan (MCP) Information System Review Worksheet & Interview
Guide (Worksheet A.2) to conduct interviews with MCP staff who completed the ISCA tool (Worksheet A.1), as well
as other MCP staff as needed. Worksheet A.2 is organized in an open-ended format by section to correspond to the
ICSA tool completed by the MCP. 85
Before the site visit with the MCP, EQRO staff should:
•

Review the ISCA Worksheet A.1 and attached documentation submitted by the MCP, including documentation
referenced in the Summary of Requested Documentation Checklist submitted with the ISCA tool.

•

Identify issues to address in follow-up interviews with MCP personnel and record the questions in this worksheet.
Revise prompts in Sections 1 through 5 as needed.

•

If the MCP’s information system has been formally assessed within the past 2 years, please review the copy of
the assessment report included with Worksheet A. Follow-up on only those sections of the assessment report that
are not covered or that may have changed since the formal assessment was conducted.

During the site visit, EQRO staff should:
•

Use the space in this Worksheet to record responses or document specific issues. It is not necessary to cover
every question in the ISCA Worksheet A.1 submitted by the MCP if responses are clear.

•

Revise this Worksheet, as needed, to provide additional space under each question to record issues and findings.

After the site visit, EQRO staff should:
Analyze findings from the ISCA Worksheet A.1 and this Worksheet and prepare a statement of findings about the
MCP’s information system.

•

Contact Information
Please insert or verify the MCP contact information below, including the MCP name, MCP contact name and title,
mailing address, telephone and fax numbers, E-mail address, and date of interview, if applicable.
MCP Name:
Contact Name:
Title:
Mailing address:
Phone number:
E-mail address:
Interview Date:
Type of delivery
system (check all that
apply)

□ MCO □ PIHP □ PAHP □ LTSS
□ Other: specify ___________________________________

Programs (please
check)

□ Medicaid (Title XIX only) □ CHIP (Title XXI only) □ Medicaid and CHIP

For the purposes of the Appendix A worksheets, the term MCP includes managed care organizations (MCOs), prepaid
inpatient health plans (PIHPs), and prepaid ambulatory health plans (PAHPs), but does not include PCCM entities, because
42 C.F.R. 438.242, which is the basis for the requirement that states ensure maintenance of health information systems, is
only applicable to MCOs, PIHPs and PAHPs.

328 | APPENDIX A

Section 1. Background Information
List questions for discussion with the MCP
Potential prompts:
○ Managed Care Model
○ Year MCP was incorporated
○ Member enrollment
○ Formal information system capability assessment
○ Recent information system enhancements

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | 329

Section 2. Information Systems: Data Processing Procedures & Personnel
List questions for discussion with the MCP
Potential prompts:
○ System or repository for Medicaid claims and encounter data
○ Programming language(s) to create Medicaid data extracts or analytic reports
○ Programmer training, time, experience, turnover
○ Standard software development methodology
○ Version control software

MCP responses to follow-up questions

Additional information provided by the MCP

330 | APPENDIX A

Section 3. Staffing
List questions for discussion with the MCP
Potential prompts:
○ Staffing productivity
○ Processor training

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | 331

Section 4. Security
List questions for discussion with the MCP
Potential prompts:
○ Disaster recovery plan
○ Disaster recovery system
○ Testing, backup systems, and storage
○ Computer system security
○ Cloud-based security
○ System access

MCP responses to follow-up questions

Additional information provided by the MCP

332 | APPENDIX A

Section 5A. Data Acquisition Capabilities: Administrative Data
List questions for discussion with the MCP about administrative data (claims and encounters)
Potential prompts:
○ Data submission methods
○ Claims or encounter submissions
○ Claims or encounter types
○ Diagnoses and procedures
○ Principal and secondary diagnoses
○ Missing, incomplete, or invalid claim/encounter submission fields
○ Claim/encounter accuracy verification
○ Systems changes/updates
○ Medicaid and CHIP data retention
○ Medicaid and CHIP claim/encounter audit policy
○ Pended claims/encounters process and reconciliation
○ Claim handling and processes that precede adjudication
○ Performance monitoring standards for Medicaid claims/encounters and results

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | 333

Section 5B. Data Acquisition Capabilities: Enrollment System
List questions for discussion with the MCP about the enrollment system
Potential prompts:
○ Changes/updates to the Medicaid enrollment data system
○ Changes/updates effect on data quality/completeness
○ Continuity of enrollee ID numbers after data system changes
○ Continuity of enrollee ID numbers if disenrolled and reenrolled
○ Linkage of claim/encounter data for Medicare/Medicaid dual eligibles
○ If using a traditional PCCM, PCCM data follows T-MSIS coding guidance
○ Timeliness of Medicaid and CHIP enrollment updates
○ Medicaid continuous enrollment

MCP responses to follow-up questions

Additional information provided by the MCP

334 | APPENDIX A

Section 5C. Data Acquisition Capabilities: Ancillary Systems
List questions for discussion with the MCP about ancillary systems
Potential prompts:
○ Use of vendor data to calculate Medicaid and CHIP quality measures
○ Subcontracted Medicaid or CHIP benefits adjudicated through a vendor's system
○ Quality and accuracy of vendor data
○ Use of vendor data in Medicaid or CHIP-related studies
○ Use of unilateral or bi-directional data linkages to health information exchanges, registries, state vital
statistics, public health data
o How the MCP verifies the data and data exchange process for these data sources

MCP responses to follow-up questions

Additional information provided by the MCP

APPENDIX A | 335

Section 5D. Data Acquisition Capabilities: Integration and Control of Data for Performance
Measure Reporting
List questions for discussion with the MCP integration and control of data for performance measure
reporting
Potential prompts:
○ MCP integration of Medicaid and CHIP claims, encounter, membership, provider, vendor, and other data to
calculate performance rates
○ Consolidation of data sets for each Medicaid and CHIP measure collected
○ Process for consolidating claims/encounter, member, and provider data for Medicaid and CHIP
○ Performance measure reporting
○ Performance measure repository
○ Report generation
○ Bonuses and penalties
○ Bundled/global payments
○ Provider profiles/directories
○ Process for maintaining Medicaid fee schedules and provider compensation rules

MCP responses to follow-up questions

Additional information provided by the MCP

END OF WORKSHEET A.2

336 | APPENDIX A

APPENDIX B. SAMPLING APPROACHES FOR
EQR DATA COLLECTION ACTIVITIES
BACKGROUND
Sampling is used frequently in EQR-related activity processes for
validation and analysis purposes, such as:
Validating performance improvement projects (PIPs) (Protocol 1)

•

Validating the performance measures included in managed care
plans’ (MCPs’) quality assessment and performance improvement
(QAPI) programs (Protocol 2)

•

Validating the encounter data reported by the MCP (Protocol 5)

•

Administering or validating quality of care surveys (Protocol 6)

•

Calculating additional performance measures (Protocol 7)

•

Implementing additional PIPs (Protocol 8)

•

Conducting focus studies of health care quality (Protocol 9)

This appendix provides a brief overview of the types of sampling
approaches and guidance for determining minimum sample sizes for
EQR data collection activities. A statistician or other staff with expertise in
sample design and implementation should advise the EQRO on the
appropriate sampling strategy to be used for each activity.

TYPES OF SAMPLING APPROACHES
Probability Sampling

Probability (or random) sampling methods leave selection of population
units to chance and not to convenience or preference on the part of the
individuals conducting the study or otherwise participating in the study.
Probability sampling removes systematic bias due to observed and
unobserved differences in the sampling units. There are several types of
probability sampling methods:
•

APPENDIX B

•

Simple random sampling is a method where all members of the
study population are listed in the sampling frame and have an equal
chance of being selected for the sample (See box, What is a
sampling frame?, next page). One way to select a simple random
sample is to first assign all units in the sampling frame a unique
identifier. Next, random numbers are generated for each unit using
random number generators (available in statistical software or
products). The random numbers then dictate the order in which units
from the sampling frame appear. Units are selected for the sample
taking the first n units in that random order, where n is the desired

337

sample size. Simple random sampling ensures that all
members of the target population have an equal
chance of selection

What is a Sampling Frame?
A sampling frame is the list from
which the sample is drawn. It
includes the universe of members of
the target study population, such as
individuals, households, providers,
or other population units that are
eligible to be included in the study.
The completeness and accuracy of
the sampling frame are key to the
representativeness of the sample.

•

Systematic random sampling is a method where
units are systematically selected starting with a
randomly selected first unit. Systematic sampling can
be used when a sampling frame is organized or
ordered in a way that does not bias the sample. Bias
can occur if, for example, there is a cyclical or seasonal
order to the data that happens to coincide with the
sampling interval, in which case the sample will not fully
represent the sampling frame. To select a systematic
sample, first determine what the sampling interval (i) is by dividing the total units in the
sampling frame (N) by the number of units in the sample (n). For example, if there are 250
units in the sampling frame and the desired sample size is 25, then the sampling interval (i)
is 250/25 = 10. Use a random number generator to select a number (k) between 1 and i.
Then select the kth, (k + i)th, (k + 2i)th, etc. units from the frame until the end of the
sampling frame is reached and you have selected n units

•

Stratified random sampling controls the proportion of the sample from subgroups of the
target population called strata. This technique divides the population into specific strata or
subgroups where the units are, ideally, homogeneous (the same or similar) within a stratum
and heterogeneous (different) between strata with respect to certain characteristics (e.g.,
age, ethnicity, or diagnosis). Stratified sampling requires weighting the sample when a
disproportionately larger number of units may be selected from one strata compared to
others (“oversampling”). Stratification is done both to improve the representativeness of the
total population’s characteristics and to provide information about the characteristics of
interest within subgroups. Stratification can be used to oversample certain subgroups or
simply to ensure that the sample ends up with the same proportion as the population with
respect to these subgroups. Once strata are identified and constructed, sampling must be
conducted within each strata, independently, using probability sampling

•

One-stage cluster sampling is used when a comprehensive sampling frame of all units is
not readily available or would be too much of a burden to construct, or when data collection
cannot occur across the entire population due to financial or operational constraints. Units
in the population are gathered or classified into groups called clusters (these groups are
similar to strata used for stratified random sampling). Unlike the stratified sampling method,
the groups ideally should be heterogeneous with respect to the measured characteristic
(but rarely are). And, unlike stratified sampling, once clusters are identified, a random
sample of clusters is selected for data collection, with data then collected from all units in
the selected clusters

•

Two-stage cluster sampling is an adaptation of one-stage cluster sampling. As with onestage cluster sampling, a sample of clusters is selected. However, unlike one-stage cluster
sampling, within the clusters there is a second stage of sampling—units within the clusters
are randomly selected so that some but not all units are selected for data collection. Twostage cluster sampling is ideal for situations where you do not have or are unable to
construct a frame of all the units in the population and you also cannot collect data from all
clusters nor all units in the selected clusters due to financial, operational, or other
constraints

338 | APPENDIX B

Non-Probability Sampling

Non-probability sampling methods are used when subjects are scarce or hard to sample (no
sampling frame) and/or the study relies on volunteers. The sample is based on the choice of
those administering the survey rather than chance; therefore, some bias can be expected. The
following are types of non-probability sampling:
•

Convenience sampling includes sampled units that are readily available or convenient to
sample. An example of a convenience sample for a focus study on patient experience with
Medicaid providers could include all patients sitting in the waiting room in a primary care
office on any given day. As another example, a focus study on health behaviors could
involve approaching people at a shopping mall

•

Quota sampling includes sampled units with known characteristics in the same proportion
as in the population. For example, if a target population is 55 percent female and 45
percent male, the quota sample requires a similar female/male distribution. Quota sampling
is considered a non-probabilistic version of a stratified sample, in which a population is
segmented into mutually exclusive subgroups and judgment is used to select a sample
based on a specified proportion

Though non-random sampling methods may be statistically analyzed, caution should be
exercised when making inferences to the study population because the sample was not drawn
randomly and therefore, may not be representative of the population. Considering the risk of
biased results and the challenges to statistical interpretation, non-probability sampling is
discouraged. However, at times, it can be an appropriate and efficient way of collecting needed
information.

CALCULATING MINIMUM SAMPLE SIZES FOR EQR DATA COLLECTION
ACTIVITIES
Many EQR-related activities may involve sampling for data collection, such as validating the
completeness and accuracy of encounter data, assessing the reliability and validity of
performance measures calculated using the hybrid method, and implementing a survey.
Statistical power is a function of the sample size, the statistical significance criterion, and the
magnitude of the effect in the population.
Table B.1 provides guidance for determining minimum sample sizes for EQR data collection
activities. The minimum sample sizes vary based on the magnitude of the proportion (or
percentage) of the effect of interest. EQROs may base the proportion on the current year’s
administrative rate or the prior year’s reported rate (column 1). When the rate is unknown,
researchers typically base the sample size on a proportion of 0.50. Researchers also typically
use a statistical significance criterion of p < 0.05, As shown in Table B.1, the minimum sample
size for a rate of 0.05 or 0.95 (5 percent and 95 percent) is 100, while the minimum sample size
for a rate of 0.50 (50 percent) is 411 (column 2). The 95 percent confidence interval (and
corresponding lower and upper bounds) indicates the range in which the true value is estimated
to lie.

APPENDIX B | 339

Table B.1. Guidance for Minimum Sample Sizes for EQR Data Collection Activities
Minimum sample size for EQR data
collection activities

95 percent confidence interval (lower
and upper bound)

0.05

100

0.043
(0.007-0.093)

0.10

159

0.047
(0.053-0.147)

0.15

219

0.047
(0.103-0.197)

0.20

270

0.048
(0.152-0.248)

0.25

313

0.048
(0.202-0.298)

0.30

348

0.048
(0.252-0.348)

0.35

380

0.048
(0.302-0.398)

0.40

398

0.048
(0.352-0.448)

0.45

409

0.048
(0.402-0.498)

0.50

411

0.048
(0.452-0.548)

0.55

409

0.048
(0.502-0.598)

0.60

398

0.048
(0.552-0.648)

0.65

380

0.048
(0.602-0.698)

0.70

348

0.048
(0.652-0.748)

0.75

313

0.048
(0.702-0.798)

0.80

270

0.048
(0.752-0.848)

0.85

219

0.047
(0.803-0.897)

0.90

159

0.047
(0.853-0.947)

0.95

100

0.043
(0.907-0.993)

Proportion

340 | APPENDIX B

DOCUMENTING SAMPLING METHODS FOR EQR DATA COLLECTION
ACTIVITIES
In general, the following information should be documented about sampling approaches used
for EQR data collection activities:
•

Definition of the population included in the data collection activity (such as denominator for
performance measure, target population for PIP or focus study, time frame for
measurement)

•

Sampling approach (such as simple random sampling, systematic random sampling,
stratified random sampling, one-stage cluster sampling, two-stage cluster sampling,
convenience sampling, quota sampling)

•

Sampling frame (such as enrollment file, claims extract, patient roster)

•

Sample exclusions (if any)

•

Sample size (including method used to determine minimum sample size)

•

Potential biases and selection issues that may affect representativeness of the sample and
generalizability of the results

APPENDIX B | 341

This page left blank for double-sided copying.

342 | APPENDIX B

APPENDIX C. ACRONYMS USED IN THE
PROTOCOLS
American Academy of Family Physicians

AHRQ

Agency for Healthcare Research and Quality

CAHPS®

Consumer Assessment of Healthcare Providers and Systems

CHIP

Children’s Health Insurance Program

CHIPRA

Children’s Health Insurance Program Reauthorization Act of 2009

CMS

Centers for Medicare & Medicaid Services

CPT®

Current Procedural Terminology

DBMS

Database Management System

DOB

Date of Birth

Disaster Recovery

DSM-IV

Diagnostic and Statistical Manual of Mental Disorders

EHR

Electronic Health Record

EQR

External Quality Review

EQRO

External Quality Review Organization

EVV

Electronic Visit Verification

FFP

Federal Financial Participation

FFS

Fee-For-Service

HCBS

Home and Community Based Services

HCPCS

Healthcare Common Procedure Coding System

HEDIS®

Healthcare Effectiveness Data and Information Set

HHS

U.S. Department of Health & Human Services

HIE

Health Information Exchange

HIPAA

Health Insurance Portability and Accountability Act

HIT

Health Information Technology

HITECH

Health Information Technology for Economic and Clinical Health Act

HPV

Human Papillomavirus Vaccine

ICD-10

International Statistical Classification of Diseases and Related
Health Problems

Information System

ISCA

Information Systems Capability Assessment

LOINC

Logical Observation Identifier Names and Codes

LOS

Length of Stay

APPENDIX C

AAFP

343

LTSS

Long-Term Services and Supports

MACBIS

CMS Medicaid and CHIP Business Information System

MCO

Managed Care Organization

MCP

Managed Care Plan

MGMA

Medical Group Management Association

MHSIP

Mental Health Statistics Improvement Program

MLTSS

Managed Long-Term Services and Supports

MRR

Medical Record Review

MSIS

Medicaid Statistical Information System

NCPDP

National Council for Prescription Drug Programs

NCQA

National Committee for Quality Assurance

NQS

National Quality Strategy

PAHP

Prepaid Ambulatory Health Plan

PCCM-E

Primary Care Case Management Entity

PCMH-A

Patient Centered Medical Home Assessment

PCP

Primary Care Provider

PDSA

Plan Do Study Act

PHI

Protected Health Information

PIHP

Prepaid Inpatient Health Plan

PII

Personally Identifiable Information

PIP

Performance Improvement Project

Quality Assurance

QAPI

Quality Assessment and Performance Improvement

Quality Improvement

Quality Strategy

T-MSIS

Transformed Medicaid Statistical Information System

344 | APPENDIX C

APPENDIX D. EXTERNAL QUALITY REVIEW
GLOSSARY OF TERMS
Acceptable Error Rate
The maximum percentage of missing, surplus, or erroneous records that
the state accepts.
Algorithm
A specific set of instructions for carrying out a procedure or solving a
problem.

Binary Variable
A discrete variable with only two categories.
Categorical Variable
A non-numeric variable with a range of non-ordered, qualitative values
(or categories). The values may be coded as numbers but should not be
interpreted numerically.
Children’s Health Insurance Program Reauthorization Act of 2009
(CHIPRA)
Reauthorized the Children’s Health Insurance Program (CHIP) under
Title XXI of the Social Security Act. CHIPRA included provisions to
strengthen the quality of care provided to children and improve health
outcomes of children in Medicaid and CHIP. CHIPRA requires the U.S.
Department of Health & Human Services (HHS) to identify and publish a
core measure set of children’s health care quality measures for voluntary
use by state Medicaid and CHIP programs (CMS Core Set of Children’s
Health Care Quality Measures for Medicaid and the Children’s Health
Insurance Program (CHIP) (the Child Core Set) and the Core Set of
Health Care Quality Measures for Adults Enrolled in Medicaid (the Adult
Core Set). The Child Core Set includes a range of children’s quality
measures encompassing both physical and mental health. The initial
Child Core Set was released in 2010, updated in 2013, and is updated
annually thereafter.

APPENDIX D

Bias
A systematic distortion in data collection, analysis, or reporting of
research findings.

Claims Data
See “Encounter Data.”
Compliance Review
A process to determine the extent to which Medicaid and CHIP managed
care plans (MCPs) are complying with the Medicaid standards set forth
at 42 C.F.R. § 438, subpart D and 42 C.F.R. § 438.330, which are
adopted by CHIP at 42 C.F.R. 457.

345

Confidence Level
The likelihood, expressed as a percentage, that percentage that a sample finding is true for
the population from which the sample was taken. For example, a 95 percent confidence
interval indicates a 5 percent chance that the sample result is due to chance and is not true
for the population.
Consumer Assessment of Healthcare Providers and Systems (CAHPS®)
A series of consumer and patient surveys rating health care experiences in the U.S. All
surveys officially designated as CAHPS® surveys have been approved by the CAHPS®
Consortium, which is overseen by the Agency for Healthcare Research and Quality (AHRQ).
CAHPS® surveys are an integral part of CMS’ efforts to improve health care in the U.S.
CAHPS® surveys follow scientific principles in survey design and development, are designed
to reliably assess the experiences of a large sample of patients, and use standardized
questions and data collection protocols to ensure that information can be compared across
health care settings.
Continuous Variable
A numeric variable with a range of numerical values. Data collected for a continuous variable
may be recoded as a discrete variable.
Correlation Coefficient
A statistical measure of the interdependence of two random variables, the value of which
indicates how much a change in one variable is related to a change in the other variable.
Correlation coefficients range in value from -1 to +1. A perfect positive correlation is +1 and a
perfect negative correlation is -1. Zero indicates the absence of a relationship between the
variables.
CPT®
A coding system, defined in the American Medical Association publication “Current Procedural
Terminology”, for medical procedures that are used for billing and quality measures.
Database Management System (DBMS)
System software for creating, managing, and maintaining databases.
Denominator
The bottom part of the fraction that represents the total number of parts created from the
whole. For the purposes of these protocols, the denominator provides the general
specifications of any clinical component that is the basis for inclusions and exclusions in the
population to be considered in a measure.
Discrete Variable
A numeric variable with a limited number of possible categories. A binary variable is a type of
discrete variable with only two categories.
Edit Checks
A program instruction that tests the quality and validity of data entered.

346 | APPENDIX D

Encounter Data
The managed care equivalent of fee-for-service (FFS) claims. Encounter data is the
information related to the receipt of any item or service by a beneficiary enrolled in a managed
care plan (MCP). They reflect that a provider rendered a specified service under a managed
care delivery system, regardless of if or how the MCP ultimately reimbursed the provider.
Encounter data include substantially the same information included on claim forms (e.g., UB04 or CMS 1500), although not necessarily in the same format. Providers submit claims or
encounters to MCPs for service(s) rendered that would traditionally be submitted as claims in
a FFS system.
Enrollee
An eligible individual who is covered by a managed healthcare plan. A beneficiary is an
eligible individual who receives health care insurance through the Medicare or Medicaid
programs.
EQR-Related Activities
The activities addressed in these protocols. EQR-related activities may be conducted by the
state, its agent that is not an MCO, PIHP, PAHP, or PCCM entity (described at 42 C.F.R. §
438.310(c)(2)), or an EQRO. See 42 C.F.R. § 438.358.
Erroneous Encounters
Encounters that occurred and are represented by an encounter record that contains incorrect
data elements.
External Quality Review (EQR)
The analysis and evaluation by an external quality review organization (EQRO), of aggregated
information on quality, timeliness, and access to the health services that an MCO, PIHP,
PAHP, or PCCM entity (described at 42 C.F.R. § 438.310(c)(2)), or their contractors furnish to
Medicaid beneficiaries.
External Quality Review Organization (EQRO)
An organization that meets the competence and independence requirements set forth at 42
C.F.R. § 438.354, and performs external quality review or other EQR-related activities as set
forth in 42 C.F.R. § 438.358, or both.
Fee-for-Service
A payment mechanism in which payment is made for each service used.
Focus Study
A study of a particular aspect of clinical care or nonclinical services provided by a managed
care plan (MCP) at a point in time. See 42 C.F.R. § 438.358(c)(5).
Generalizability
The extension of findings and conclusions from a study sample to the population from which
the sample was drawn.
Healthcare Common Procedural Terminology (HCPCS)
A standardized coding system for describing the specific items and services provided in the
delivery of health care. HCPCS contains levels of codes, including the American Medical
Association’s CPT® and alphanumeric codes for non-physician services, items, and supplies
not contained in CPT®.

APPENDIX D | 347

Healthcare Effectiveness Data and Information Set (HEDIS®)
A collection of standardized performance measures and their definitions designed to ensure
that purchasers and consumers can reliably compare the performance of managed health
care plans. The performance measures are related to public health issues such as cancer,
heart disease, and asthma and also include well-child visits. HEDIS® is sponsored,
supported, and maintained by the National Committee for Quality Assurance (NCQA).
Health Information Technology (HIT)
Used by health care providers to manage patient care and health through using and sharing
health information in a secure system. EHR, meaningful use, and mobile health laws and
regulations all fall under the umbrellas of HIT.
Hybrid Data
Administrative data supplemented by medical record review.
Improvement Strategy
An intervention designed to change behavior at the member, provider, and/or managed care
plan (MCP)/system level.
Indicator
An observable and measurable characteristic that can be used to show changes or progress
over time toward achieving a specific outcome.
Information System Capabilities Assessment (ISCA)
Assessment of the desired capabilities of the MCP’s information system which poses standard
questions used to assess the strength of the system; this provides information to the EQRO
about the extent to which the information system is capable of producing valid encounter data,
performance measures, and other data necessary to support quality assessment and
improvement, as well as managing the care delivered to its beneficiaries. Please refer to
Appendix A. Information System Capabilities Assessment for more information.
Kappa statistic
A test statistic that measures interrater reliability for categorical data (e.g., sex, gender, race,
etc.).
Locating
Locating is a technique used to improve response rates by locating and contacting sample
members. This includes verified collection of data, such as first and last name, home address,
email address, phone number(s), date of birth, language preference, etc.
Managed Care Plans (MCPs)
For the purposes of the EQR protocols, encompasses managed care organizations (MCOs),
prepaid inpatient health plans (PIHPs), prepaid ambulatory health plans (PAHPs), and the
subset of primary care case management (PCCM) entities described in 42 C.F.R. §
438.310(c)(2).
Managed Care Quality Strategy
See “State Quality Strategy.”

348 | APPENDIX D

Margin of Error
A statistic expressing the amount of random sampling error in a survey's results. The larger
the margin of error, the less faith one should have that the sample result is the true population
value.
Measure
A standard used for valuing or determining the extent or quantity of something.
Missing Encounters
Encounters that occurred but are not represented by an encounter record.
Non-Probability Sampling
Methods that are used when subjects are scarce or hard to sample (no sampling frame)
and/or the study relies on volunteers. The sample is based on the choice of those
administering the survey rather than chance; therefore, some bias can be expected. Nonprobability sampling includes convenience sampling and quota sampling. Please refer to
Appendix B. Sampling Approaches for EQR Data Collection Activities for more information.
Numerator
The top part of the fraction that represents how many parts of that whole are being
considered. For example, with large population of patients, the numerator would be the
number of patients in a study meeting the specifications of a clinical component in a measure.
Pay for Performance
An umbrella term for initiatives aimed at improving the quality, efficiency, and overall value of
health care. These arrangements provide financial incentives to hospitals, physicians, and
other health care providers for improvements in quality of care and health outcomes for
patients.
Pearson Correlation Coefficient (also, Pearson’s r)
The most common measure of correlation in statistics. It shows the linear relationship
between two variables X and Y, with results between -1 and +1, where 1 is total positive linear
correlation, 0 is no linear correlation, and -1 is negative linear correlation. The closer the value
to zero, the greater the variation the data points are around the line of best fit.
Performance Improvement Project (PIP)
A project that implements an intervention designed to achieve and sustain significant
improvement in health outcomes over time.
Performance Measure
Used to monitor performance at a point in time, track performance over time, compare
performance, and inform decisions. For the purposes of these protocols, it refers to monitoring
the performance of individual managed care plans (MCPs) at a point in time, to track MCP
performance over time, to compare performance among MCPs, and to inform the selection
and evaluation of quality improvement activities.
Plan Do Study Act (PDSA)
A continuous cycle of measuring and analyzing performance. For the purposes of these
protocols, PDSA cycles refer to testing changes on a small scale and applying rapid-cycle
learning principles to adjust intervention strategies over the course of time (such as in PIPs).

APPENDIX D | 349

Prepaid Ambulatory Health Plan (PAHP)
An entity that provides services to enrollees under contract with the state and on the basis of
capitation payments or other payment arrangement’s that do not use state plan payment
rates; does not provide or arrange for and is not otherwise responsible for the provision of any
inpatient hospital or institutional services for its enrollees; and does not have a comprehensive
risk contract.
Prepaid Inpatient Health Plan (PIHP)
A prepaid health plan that provides services to enrollees under contract with the state and on
the basis of capitation payments or other payment arrangements that do not use State plan
payment rates; provides, arranges for, or otherwise has responsibility for the provision of any
inpatient hospital or institutional services for its enrollees; and does not have a comprehensive
risk contract.
Primary Care Case Management (PCCM)
A system under which a primary care case manager contracts with the state to furnish case
management services (which include the location, coordination and monitoring of primary
health care services) to Medicaid beneficiaries.
Primary Care Case Management (PCCM) Entity
The term PCCM entity in these EQR protocols only applies to those PCCM entities whose
contracts with a state provide for shared savings, incentive payments, or other financial
reward for the PCCM entity for improved quality outcomes, as described at 42 C.F.R. §
438.310(c)(2).
Probability (or random) Sampling
Refers to sampling methods that leave selection of population units to chance and not to
convenience or preference on the part of the individuals conducting the study or otherwise
participating in the study. Probability sampling removes systematic bias in the selected
sample due to observed and unobserved differences in the sampling units. Types of
probability sampling include simple random sampling, systematic random sampling, stratified
random sampling, one-stage cluster sampling, and two-stage cluster sampling. Please refer to
Appendix B. Sampling Approaches for EQR Data Collection Activities for more information.
Programmatic Significance
The practical effect or importance of an intervention implemented through a program or
specified method.
Protected Health Information
A class of patient data that can be linked to a specific individual.
Quality
The degree to which an MCO, PIHP, PAHP, or PCCM entity (described at 42 C.F.R. §
438.310(c)(2)) increases the likelihood of desired health outcomes of its enrollees through
structural and operational characteristics, the provision of services that are consistent with
current professional, evidence-based knowledge, and interventions for performance
improvement.

350 | APPENDIX D

Quality Assurance Plan
A plan that includes processes to monitor, evaluate and review all aspects of the survey
administration procedures. The purpose of a quality assurance plan is to document reviews
and audits to ensure appropriate processes are correctly followed.
Registry Data
Clinical data that is recorded about the health status of patients and health care they receive
over time. This data is maintained in a clinical data registry.
Reliability
Refers to (1) the internal consistency of a study instrument, and (2) that data are producing
consistent results.
Sample
A subset selected from a population.
Sampling Frame
The list from which the sample is drawn. It includes the universe of members of the target
study population, such as individuals, households, encounters, providers, or other population
units that are eligible to be included in the study. The completeness, recency, and accuracy of
the sampling frame are key to the representativeness of the sample.
Significant Improvement
A measurable, statistically significant change in performance related to an intervention.
State Quality Strategy
A strategy to assess and improve the quality of Medicaid managed care services within a
state, per 42 C.F.R. § 438.340 and adopted by CHIP at 42 C.F.R. § 457.1240(e).
Statistical Significance
A measure of whether research findings are meaningful. More specifically, whether results
match closely to what one would expect to find in an entire population. The test for statistical
significance requires (1) deciding an alpha level, meaning, the error rate (typically 5 percent or
less), (2) collecting data, (3) calculating the test statistic, and (4) comparing the calculated test
statistic with a statistic from a statistical table.
Study Population
The population identified for the study. It may include the entire population or a sample of the
population depending on the nature of the study question and available data.
Study Question
Identifies the focus of the study and sets the framework for data collection and analysis. The
study question should be clear, concise, and answerable.
Study Variable
A measurable characteristic, quality, trait, or attribute of a particular individual, object, or
situation being studied.

APPENDIX D | 351

Sustained Improvement
Significant changes in processes or performance as demonstrated through repeated
measurements over comparable time periods using the same methodology as in the baseline
measurement.
Target Population
The group of individuals that are the intended recipient of a particular service or intervention.
Transformed Medicaid Statistical Information System (T-MSIS)
A critical data and systems component of the CMS Medicaid and CHIP Business Information
System (MACBIS). CMS has been working with states to transform the MSIS system, which
was used to (1) collect utilization and claims data as well as other key Medicaid and CHIP
program information, (2) keep pace with the data needed to improve beneficiary quality of
care, (3) assess beneficiary care and enrollment, (4) improve program integrity, and (5)
support states, the private market, and stakeholders with key information. The T-MSIS data
set contains (1) enhanced information about beneficiary eligibility, (2) beneficiary and provider
enrollment, (3) service utilization, (4) claims and managed care data, and (5) expenditure data
for Medicaid and CHIP.
T-test
Most commonly used with small sample sizes, this test asks whether a difference between
two samples/groups’ averages is unlikely to have occurred because of random chance in
sample selection. A difference is more likely to be meaningful or if (1) the difference between
the averages is large, (2) the sample size is large, and (3) the standard deviation is low.
Unit of Analysis
The entity (“what” or “whom”) that is being studied.
Validation
The review of information, data, and procedures to determine the extent to which it is
accurate, reliable, free from bias, and meets standards for data collection.
Validity
The degree to which a tool measures what it is intended to measure.
Verification
The internal review of documentation, data, measures, and assessments to determine if
measurements are accurate.
Vital Records
Records of life events kept under government authority. These include life events such as
birth certificates, marriage licenses, and death certificates.

352 | APPENDIX D

File Type	application/pdf
File Title	CMS External Quality Review (EQR) Protocols
Subject	EQR, EQRO, Protocols, Medicaid, Managed Care, CHIP, CMS, Quality, Medicaid managed care rule
Author	Centers for Medicare & Medicaid Services
File Modified	2019-07-25
File Created	2018-09-05