External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOL 1:
ASSESSMENT OF COMPLIANCE
WITH MEDICAID MANAGED CARE
REGULATIONS
A Mandatory Protocol for External Quality Review (EQR)
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations
Protocol 2: Validation of Measures Reported by the MCO
Protocol 3: Validation of Performance Improvement Projects (PIPs)
Protocol 4: Validation of Encounter Data Reported by the MCO
Protocol 5: Validation and Implementation of Surveys
Protocol 6: Calculation of Performance Measures
Protocol 7: Implementation of Performance Improvement Projects (PIPs)
Protocol 8: Focused Studies
Appendix V: Information Systems Capabilities Assessment (ISCA)

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Protocol 1
Version 2.0
December 1, 2011

Table of Contents
PURPOSE AND OVERVIEW OF THE PROTOCOL ....................................................... 3
ACTIVITY 1: ESTABLISH COMPLIANCE THRESHOLDS ............................................. 4
Step 1: Collect Information from the State ................................................................. 4
Step 2: Define Levels of Compliance ......................................................................... 4
ACTIVITY 2: PERFORM PRELIMINARY REVIEW ......................................................... 4
Step 1: Establish Early Contact with the MCO ........................................................... 4
Step 2: Perform a Preliminary Document Review ...................................................... 5
Step 1: Determine the Length of Visit and the Visit Dates ......................................... 6
Step 2: Identify the Number and Types of Reviewers Needed .................................. 6
Step 3: Establish an Agenda for the Visit ................................................................... 6
Step 4: Provide Preparation Instructions and Guidance to the MCO ......................... 6
Step 5: Conduct Onsite Document Review ................................................................ 7
Step 6: Conduct MCO Interviews ............................................................................... 7
Step 7: Exit MCO Interviews ...................................................................................... 9
ACTIVITY 4: COMPILE AND ANALYZE FINDINGS ....................................................... 9
Step 1: Collect Supplemental Information .................................................................. 9
Step 2: Compile Data and Information ..................................................................... 10
Step 3: Analyze Findings ......................................................................................... 10
ACTIVITY 5: REPORT RESULTS TO THE STATE ...................................................... 10
Step 1: Submit a Final Determination Report to the State ....................................... 11
Step 2: Submit Other Reports Requested by the State............................................ 11
Attachments
Attachment A: Compliance Review Worksheet
Attachment B: Compliance Definitions
Attachment C: Sample Site Visit Agenda
Attachment D: Compliance Interview Questions

EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

2

PURPOSE AND OVERVIEW OF THE PROTOCOL
This mandatory EQR protocol is used to determine, in a manner consistent with standard
industry practices, the extent to which Medicaid and CHIP MCOs are in compliance with Federal
quality standards mandated by the Balanced Budget Act of 1997 (BBA). HHS developed the
standards, which are codified in regulation at 42 C.F.R. § 438. This protocol is particularly
important after the passage of the Affordable Care Act and the creation of new entities such as
Accountable Care Organizations, Health Homes, Affordable Insurance Exchanges, and Health
Information Exchanges.
Protocol 1 describes a process States or their contractors may use every 3 years to determine
MCO compliance with Federal Medicaid managed care regulations. Protocol 1 describes how to
review program documents and conduct interviews with MCO personnel to efficiently collect the
information necessary to determine compliance with regulations at 42 C.F.R. § 438, subparts D
and E. In addition, Protocol 1 provides options for analyzing information and making compliance
determinations.
Although document review and interviews are the primary activities associated with Protocol 1,
additional activities are necessary to prepare for, effectively support, and conclude the
compliance determination. Protocol 1 activities are presented in the order in which they should
be undertaken. The five activities that comprise this protocol are:
1.
2.
3.
4.
5.

Establish Compliance Thresholds;
Perform Preliminary Review;
Conduct MCO Site Visit;
Compile and Analyze Findings; and
Report Results to the State.

This protocol includes four tools to use in performing the protocol activities:
1. A compliance review worksheet (Protocol 1, Attachment A);
2. Compliance definitions (Protocol 1, Attachment B);
3. A sample site visit agenda (Protocol 1, Attachment C); and
Compliance review interview questions (Protocol 1, Attachment C).
Results of the MCO’s compliance review may be reported in the annual Secretary’s Report on
the Quality of Care for Children in Medicaid and CHIP or the annual Secretary’s Report on the
Quality of Care for Adults in Medicaid. These reports are released every September and
information that is not available from a State’s EQR report may be so noted in the reports. Both
reports will be available on the CMS Medicaid website. CMS strongly encourages States to
have final EQR Technical Reports available to CMS and the public by April of each year, for
data collected within the prior 15 months. This submission timeframe aligns with the collection
and annual reporting on managed care data by the Secretary each September 30th, as required
under the Children’s Health Insurance Program Reauthorization Act (CHIPRA) [Sec. 401 (c)(2)]
and the Affordable Care Act [Sec. 2701 (d)(2)].

EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

3

Please note that a qualified EQRO must draft a final report for States that conduct their own
assessment of compliance with Medicaid managed care regulations (Protocol 1) in accordance
with the requirements of 42 C.F.R. § 438.356.

ACTIVITY 1: ESTABLISH COMPLIANCE THRESHOLDS
This activity provides clear definitions and rules for compliance scoring. Pre-planning for the use
of review results is essential to implementation of an effective assessment. The EQRO will work
with the State to define levels of compliance for use throughout the compliance review.

Step 1: Collect Information from the State
Some regulatory provisions allow States to establish more stringent standards for their MCOs
than are mandated by Federal regulation. Therefore, the EQRO will need to know the State’s
requirements for its MCOs in order to complete the compliance assessment. The Compliance
Review Worksheet provided in Attachment A includes the types of documents that the State
may provide to the EQRO about State standards and is organized according to Federal
regulatory provision. The Compliance Review Worksheet also contains individual components
of the Federal regulatory provisions so that reviewers can collect specific aspects of State
standards as needed.
States are encouraged, but not required, to contract with EQROs to provide a comprehensive,
aggregated summary report of all MCO findings. This summary report is useful in determining
whether the State has met the goals and objectives laid out in the State’s Medicaid Managed
Care State Quality Improvement Strategy.

Step 2: Define Levels of Compliance
EQRO determinations will be based on compliance definitions set in advance by the State for
each Federal and/or State regulatory provision, component of a provision, and/or requirement or
standard based on its expectations of MCO performance. While States may define multiple
levels of compliance, a definition for full compliance must be clearly understood by the EQRO
and MCO prior to the commencement of the review. Attachment B provides examples of
compliance definitions, as well as an option for defining fewer regulatory provisions upon which
to base the overall MCO compliance in order to target more important and specific State and
Federal regulatory provisions.

ACTIVITY 2: PERFORM PRELIMINARY REVIEW
Site visits are an effective way to collect the information needed for quality oversight and
compliance determination. However, they require careful planning to maximize the information
obtained and to minimize the time required for collecting that information. This activity should
begin from 2 to 6 months in advance of the planned visit.

Step 1: Establish Early Contact with the MCO

EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

4

It is important for the EQRO to establish and maintain consistent communication with the MCO
throughout the compliance review. The reviewer is responsible for developing a communication
plan specifying expectations for all parties involved. The EQRO should establish a single point
of contact with the MCO to develop appropriate accountability for clear and open
communication. The MCO representative can then organize the response from the MCO and
determine which additional staff members should be involved during the review.

Step 2: Perform a Preliminary Document Review
The purpose of a preliminary document review is to identify gaps in information necessary to
ensure a comprehensive EQR process and efficient and productive interactions with the MCO
during the site visit. Prior to planning a site visit, the reviewer should gather and assess as much
information about the MCO and its practices as possible. Preliminary document review includes
gathering information about the MCO’s background, including its structure, enrolled population,
providers, services, operations (e.g., locations where activities take place, delegated services
and contractors), resources, and delegated functions. Some information may be available from
the State, while some may be obtained from the MCO. The following list suggests the type of
information that would be useful during a preliminary document review:
a.
b.
c.
d.
e.
f.

Organization name and mailing address;
Contact person’s name, title, phone number, e-mail address;
Location for the site visit (e.g., headquarters address and service locations);
Organizational charts or other structural descriptions of the MCO;
Product lines offered;
Total individuals enrolled (for current and previous year) with a breakdown by
product line;
g. Total individuals served (for current and previous year);
h. Total number of network practitioners (for current and previous year) with a
breakdown by type (e.g., primary care, OB/GYN, and other specialties);
i. Total number of network organizational providers (hospitals, ambulatory care, home
care, laboratories, etc.);
j. Service descriptions and benefit designs available to enrollees;
k. Delegated activities; and
l. Data on MCO Quality of Care review.

Please refer to the Compliance Review Worksheet in Attachment A for a list of the types of
documents that the MCO may provide to the EQRO that may demonstrate the MCO’s
compliance with Federal regulation and State standards. The subject matter or category of each
MCO document listed in the worksheet is indicated in parentheses to assist the MCO and/or
reviewer(s) locate the document. The subject matter designation does not imply that the
document cannot be used as a data source for addressing other provisions or that it should be
the sole source of data in assessing compliance with the provisions noted.
The Compliance Review Worksheet can be used as a standardized template for recording the
documents reviewed and can provide the rationale for a final determination for each regulatory
provision or component of a provision. The completed Compliance Review Worksheet will
provide both a primary data source for analyses and a comprehensive record of EQR activities.
The State may also customize the template if needed.
EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

5

ACTIVITY 3: CONDUCT MCO SITE VISIT
The purpose of the MCO site visit is to collect the information necessary to assess the MCO’s
compliance with Federal and State regulations. The EQRO collects information through
document review, interviews, and accessory sources. The EQRO should plan the site visit in
accordance with the compliance review plan established in Activity 1 of this protocol (establish
compliance protocol). As noted in Activity 2, the EQRO should review MCO policies and
procedures prior to the site visit to expedite the process.

Step 1: Determine the Length of Visit and the Visit Dates
The time required for a comprehensive site visit will vary according to the scope of the review,
the complexity of the organization being reviewed, the number of reviewers available to conduct
the review, and the amount of information collected prior to the site visit. Based on the
experience of accrediting bodies that assess compliance with standards similar to those in the
Medicaid managed care regulations, the average length of a site visit is 3 to 5 days. To
schedule the site visit, the EQRO should offer the MCO contact a range of dates from which to
select specific days when essential staff will be available.

Step 2: Identify the Number and Types of Reviewers Needed
Reviewers should be skilled interviewers with the ability to read and process a variety of data in
order to determine whether an MCO is in compliance with the regulations. Knowledge or
experience in State CHIP/Medicaid programs and managed care are highly desirable
qualifications for reviewers. Reviewer orientation and training should be held to ensure
familiarity with the regulatory provisions, the evaluation process, and performance expectations.
The number of reviewers needed to conduct the site assessment should be based on the
characteristics of the MCO that is being evaluated. Consideration should be given to the size
and complexity of the MCO, including the size of provider network, number of enrollees, and the
scope of programs in the State contract. If multiple reviewers will be conducting the site visit(s),
the EQRO should identify in advance each reviewer’s responsibility for assessing specific
standards, reviewing specific documents, and conducting interviews.

Step 3: Establish an Agenda for the Visit
Clear expectations are essential for an efficient and effective site visit. An agenda will assist
both the MCO and EQRO in planning for staff participation, gathering documentation, and
addressing logistical issues, such as arranging locations for document review and interviews.
An agenda sets the tone, expectations, the objectives, and time frames for the review. It should
include an orientation as to how the MCO organizes its documents that will be reviewed during
the site visit. The EQRO should consult with the MCO throughout the agenda setting process to
ensure the inclusion of appropriate staff. A sample agenda format can be found in Attachment
C.

Step 4: Provide Preparation Instructions and Guidance to the MCO
EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

6

The EQRO should send clear instructions and guidance to the MCO prior to the site visit. In
preparation for the site visit, the reviewers should provide MCOs with the following information:
a.
b.
c.
d.
e.
f.

The scope of the assessment;
How the review will be conducted;
Lists of required documents;
Instructions for the organization of document presentation;
Forms or other data gathering instruments that should be completed prior to arrival
(e.g., an Information Systems Capability Assessment (ISCA) - see Appendix V);
Reports from prior reviews and subsequent MCO corrective actions;

g. Identification of expected interview participants; and
h. Administrative needs of the reviewers and any other expectations or responsibilities.

Step 5: Conduct Onsite Document Review
Prior to the site visit, the reviewers should review the standards identified in the State
documents obtained in Activity 1. MCO staff need not be present during the document review,
but should be available if reviewers have questions or difficulty locating documents or other
information. The reviewer(s) should notify the MCO during the review of any missing information
to allow the MCO to respond in a timely manner. The reviewer(s) should maintain consistent
documentation by adding onsite document review findings to the Compliance Review
Worksheet used in the preliminary document review.

Step 6: Conduct MCO Interviews
The purpose of MCO interviews is to collect data to supplement and verify what is learned
through the preliminary document review and onsite document review. The interviews should
clarify and confirm that what has been documented with what is carried out in practice.
Interview Preparation
Attachment D provides compliance review questions organized by regulatory provision and
interviewee group. Interviews should be tailored to the MCO being evaluated and the role of the
interviewee. Interview planning should include:
•
•
•

Preparing a list of issues to be addressed in each interview, based on Federal regulatory
provisions, State standards, MCO organization characteristics, and other information
gathered during pre-site document reviews;
Reviewing the MCO’s anticipated interview participants, as well as identifying topics that
will engage as many individuals in the discussion as possible;
When multiple reviewers are assigned to an MCO, the team should assign primary roles
to each reviewer (e.g., interviewer or note-taker), yet be flexible enough to allow shared
roles and responsibilities as appropriate throughout the site visit.

It is strongly recommended that the document review be completed prior to the commencement
of interviews. Some interview participants may provide documents during their interview. This
EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

7

might be done when such documents are vital to the discussion or if the review of the
documents will benefit from interactive review by all participants.
Interview Participants
Interviews should be conducted with groups, rather than with single individuals, because rarely
does one individual have sole responsibility for a particular function. Interview groups should
include participants that represent different functions, services, or departments of the MCO to
enable the reviewer to collect multiple perspectives about an issue. Group interviews are also
an opportunity for MCO staff to learn about compliance activities in other departments. The
EQRO has discretion to meet with less than the full list of MCO recommended employees in
situations where the EQRO feels that it can obtain the required information without the
attendance of all MCO employees listed in the Protocol, or the MCO has identified a more
appropriate person to address questions but is not on the recommended list. Attachment D
includes compliance review questions for the following groups:
•
•
•
•
•
•
•
•
•

MCO leaders;
MCO information systems staff;
Quality assessment and performance improvement program staff;
Provider/ contractor services staff;
Enrollee services staff, including grievance and appeal staff;
Utilization management staff;
Medical Director(s);
Case managers and care coordinators; and
MCO providers and contractors, as appropriate and as time and resources permit.

Interview Process
The reviewer should provide the MCO interview participants with an interview agenda prior to
the interview, which includes the interview goals, issues, topics, and a listing of related materials
or documents. Effective facilitation of an interview with an individual or a group requires that the
reviewer or lead interviewer:
•
•
•
•
•
•
•

Maintain control of the interview discussion by politely redirecting participants to the topic
or question as necessary;
Adhere to the time frames outlined in the agenda;
Listen carefully to participants and summarize or restate participant responses to ensure
understanding;
Take notes using the Compliance Review Worksheet or similar tool, or according to the
Compliance Review Questions provided in Attachment D;
Review documents provided during the interview at an appropriate time based on the
content and purpose of sharing the document;
Conclude the interview with a review of the outlined goals and compliance levels to
ensure an understanding of the extent to which they were met; and
Provide information about the next steps of the site , as appropriate.

Interviews & Systems
EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

8

States have the opportunity to expand the roles of other State agencies in terms of their
responsibilities related to data exchanges, EHRs, interoperability, care coordination, and
Medicaid or CHIP waivers. At the State’s discretion, it may determine:
• Whether the EQRO will review the State’s health information technology (HIT) plan for
HITECH and meaningful use with respect to validation of performance measures or
performance improvement project activities; and
• How the MCO’s systems will support State efforts in a valid way.
Also refer to Appendix V – Information Systems Capabilities Assessment

Step 7: Exit MCO Interviews
The EQRO will perform an exit interview at the conclusion of the site visit with MCO staff that
are identified by the MCO representative. The purpose of the exit interview is to clarify the
EQRO’s understanding of the information collected throughout the compliance review process.
The EQRO should provide the MCO with the opportunity to respond to initial compliance issues
to ensure the findings are due to true non-compliance and not due to misunderstanding or
misinterpretation of MCO documents and interviews.

ACTIVITY 4: COMPILE AND ANALYZE FINDINGS
The purpose of this activity is to document components of the review and the final compliance
determinations for each regulatory provision. The documented findings will serve as evidence of
the comprehensiveness of the EQR process and validity of the determinations.

Step 1: Collect Supplemental Information
In addition to information collected during the site visit, the reviewer should consider other
sources of information that confirm the MCO’s compliance with Federal regulations and State
standards. Additional sources should include the following:
• Results of Medicaid beneficiary surveys (See Protocol 5 for details about validating and
calculating surveys);
• Results of independent assessments of the MCO’s information systems (See Appendix
V for details about performing an ISCA);
• Results of independent assessments of MCO encounter data (See Protocol 4 for details
about validating encounter data);
• Results of independent validations of MCO performance measures (See Protocols 2 and
6 for validating and calculating performance measures respectively);
• Results of independent validation of performance improvement projects (PIPs) (See
Protocols 3 and 7 for details about validating and implementing PIPs respectively or
Protocol 8 for details about conducting a Focused Study); and
• Additional materials requested during or after the onsite visit, such as grievance and
appeal reports and analyses.

EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

9

Step 2: Compile Data and Information
Reviewers should document any additional information they review, including sources of the
information and their findings about the MCO’s compliance, to the Compliance Review
Worksheet used during preliminary document review and the onsite visit.

Step 3: Analyze Findings
One commonly used, but not mandatory, approach to analyzing EQR findings is to assign a
numerical value to indicate the degree of compliance with a given regulatory provision. A
reviewer can provide a compliance score for each regulatory provision on the Compliance
Review Worksheet, followed by details and justification for the compliance determination. The
determination options provided in the template include ‘Met’, ‘Not Met’, and ‘Not Applicable’.
However, these options should be adapted according to the compliance thresholds established
by the State (See Activity 1 of this protocol). States and EQROs can adapt a compliance rating
scale to best suit their needs. Some examples include:
•
•

•

Two Points
• Met
• Not Met
Three Points
• Fully Met
• Partially Met
• Not Met
Five Points
• Fully Met
• Substantially Met
• Partially Met
• Minimally Met
• Not Met

Regardless of the number of points on a scale, each level of compliance must be defined clearly
for the State, the EQRO, and the MCO prior to commencement of the review. While one scale
may serve as the primary method of assigning levels of compliance, it does not preclude the
combined use of another scale. For example, a five-point compliance scale may be appropriate
for most of the provisions, but some provisions may be dichotomous (e.g., met or not met).
When determinations are made for levels of compliance other than ‘met’ or ‘not met’, such as
‘partially met’, the reviewer should clearly identify specific deficiencies, as well as the rationale
and evidence of the deficiency.

ACTIVITY 5: REPORT RESULTS TO THE STATE
The EQRO will draft a report to the State with the results of the review of the MCO’s compliance
with Federal and State requirements. This report, based on documented reviewer
determinations and rationale, will reflect the State’s reporting needs identified prior to
commencement of the review.

EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

10

Step 1: Submit a Final Determination Report to the State
The EQRO should develop an outline that is approved by the State. Because the State may use
the report to meet its reporting requirements for Federal or State agencies, the State legislature,
local advocacy groups, as well as other interested parties, the report may need certain types of
information presented in a specific format. While non-summarized findings might be of interest
to some individuals, the report should include an overall summary of findings for compliance
with regulatory provisions.
By design, the Compliance Review Worksheet separates the regulatory provisions into four
major sections: Enrollee Rights; Access; Measurement and Improvement; and Grievance
System. Although each regulation is assigned a level of compliance, the EQRO and the State
may group select regulatory provisions together to combine ratings into one aggregate
compliance score. The degree of “roll-up” or aggregation will be determined by the level of
reporting required by the State to meet its reporting requirements or its interest in comparison
and/or MCO incentive programs (e.g., Statewide pay for performance programs).

Step 2: Submit Other Reports Requested by the State
Upon completion of an EQR, it is common practice to share the results with the MCO. The State
may request a specific format for reporting results back to the MCO. Some options for reporting
evaluation results to the MCO, include, but are not limited to:
1. Compliance Issues Only: Reviewers provide verbal feedback about general compliance
issues they have identified during the course of the EQR. Neither compliance
determinations for individual regulatory provisions nor findings for a level of MCO
performance are discussed. This type of feedback typically is provided to the MCO
leadership during a closing session or exit interview at the onsite visit. This provides the
MCO the opportunity to offer additional information, if evidence of compliance is
available.
2. Compliance Issues Specific to Regulatory Provisions: Reviewers provide verbal
feedback for regulatory provisions or components of provisions that are determined less
than fully compliant, in accordance with the compliance thresholds established by the
State prior to the review. Findings for a level of MCO performance are not discussed.
This type of feedback typically is presented to the MCO leadership during a closing
session or exit interview at the onsite visit. This provides the MCO the opportunity to
offer additional information, if evidence of compliance is available.
3. Compliance Determinations and Deficiency Report: Reviewers provide verbal and/or
written feedback about identified compliance issues, compliance ratings for regulatory
provisions, and an overall finding for MCO performance, highlighting areas of deficiency
that will be presented to the State.

END OF PROTOCOL

EQR Protocol 1:
Assessment of Compliance with Medicaid Managed Care Regulations
December 2011

11