CMS-R-305 Conducting Focus Studies of Health Care Quality

OMB Approval No.
0938-0786

CONDUCTING FOCUSED STUDIES OF
HEALTH CARE QUALITY

A protocol for use in Conducting Medicaid External Quality
Review Activities

Department of Health and Human Services
Centers for Medicare & Medicaid Services

Final Protocol
Version 1.0
May 1, 2002
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid
OMB control number. The valid OMB control number for this information collection is 0938-0786. The time required to complete this
information collection is estimated to average 1,591 hours per response for all activities, including the time to review instructions, search existing
data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the
time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer,
Baltimore, Maryland 21244-1850.
Form CMS-R-305

1

CONDUCTING FOCUSED STUDIES OF HEALTH CARE
QUALITY
I.

PURPOSE OF THE PROTOCOL

The purpose of health care quality assessment is to evaluate processes and outcomes of care. In
order for an assessment to be acceptable to interested parties, the assessment must be designed,
conducted and reported in a methodologically sound manner. Sometimes a State may want to
have a study conducted on a one-time basis and not necessarily follow-up with a repeat study for
quality improvement (QI) purposes. This protocol specifies procedures for external quality
review organizations (EQROs) 1 to use in conducting focused studies of clinical and nonclinical
health care services provided by Medicaid Managed Care Organizations (MCOs) and Prepaid
Inpatient Health Plans (PIHPs) at a point in time as directed by the State Medicaid agency. As
each step for conducting a focused study is conducted, information should be recorded on a
standardized worksheet such as that located in Attachment A.
II.

OVERVIEW OF THE PROTOCOL

This protocol has been derived from existing public and private sector tools and approaches to
conducting quality assessment and improvement projects (See Attachment B). Activities that all
public and private sector tools have in common were included in this protocol. In addition,
activities found in fewer documents were included where the activity was felt to be important to
promoting stronger focused studies, but would not result in an inappropriate burden on the MCO,
PIHP or the EQRO. In particular, the protocol relies heavily on a guidebook produced by the
National Committee for Quality Assurance (NCQA) under a contract from the Centers for
Medicare & Medicaid Services (CMS), formerly the Health Care Financing Administration
(HCFA), “Health Care Quality Improvement Studies in Managed Care Settings: A Guide for
State Medicaid Agencies” This guidebook identifies key concepts related to the conduct of QI
studies and details widely accepted principles in designing, implementing and assessing QI
studies.
This protocol describes seven steps to be undertaken when conducting focused studies:
1.
2.
3.
4.

Select the study topic(s)
Define the study question(s)
Select the study indicator(s)
Use a representative and generalizable study population

1

It is recognized that a State Medicaid agency may choose an organization other than an EQRO (as defined
in Federal regulation) to conduct MCO or PIHP focused studies. However, for convenience, in this protocol we use
the term “external quality review organization (EQRO)” to refer to any organization that conducts focused studies
for a MCO or PIHP.

2

5.
6.
7.
III.

Use sound sampling techniques (if sampling is used)
Reliably collect data
Analyze data and interpret study results

PROTOCOL ACTIVITIES

Activity 1:

Select the Study Topic(s)

Rationale. All focused studies should target improvement in relevant areas of clinical care and
non-clinical services. Topics selected for study must reflect the Medicaid enrollment in terms of
demographic characteristics, prevalence of disease and the potential consequences (risks) of the
disease. Information on Medicaid enrollees can be obtained from the following sources. Note
also that State Medicaid agencies may select the study topic.
Potential Sources of Information on Medicaid Enrollees:
-

-

-

-

-

Data in the MCO’s/PIHP’s enrollment/membership files on enrollee characteristics
relevant to health risks or utilization of clinical and non-clinical services, such as age, sex,
race/ethnicity/language and disability or functional status.
Utilization, diagnostic, and outcome information on outpatient and inpatient Medicaid
encounters, services, procedures, medications and devices, admitting and encounter
diagnoses, adverse incidents (such as deaths, avoidable admissions, or readmissions); and
patterns of referrals or authorization requests obtained from MCO/PIHP encounter,
claims, or other administrative data.
Data on the MCO’s/PIHP’s performance as reflected in standardized measures,
including, when possible: local, State, or national information on performance of
comparable organizations.
Data from other outside organizations, such as Medicaid or Medicare fee-for-service
data, data from other health plans, and local or national public health reports on
conditions or risks for specified populations.
Data from surveys, grievance and appeals processes, and disenrollments and requests to
change providers.
Data on appointments and provider networks (e.g., access, open and closed panels, and
provider language spoken).

Methods of Implementation:
In general, a clinical or non-clinical issue selected for study should affect a significant portion of
the enrollees (or a specified sub-portion of enrollees) and have a potentially significant impact on
enrollee health, functional status or satisfaction. The topics should reflect high-volume or highrisk conditions of the population served. High-risk conditions may occur for infrequent
conditions or services, such as when a pattern of unexpected adverse outcomes are identified
through data analysis. High risk also exists for populations with special health care needs, such
as

3

children in foster care, adults with disabilities and the homeless. Although these individuals may
be small in number, their special health care needs place them at high risk.
Address the following considerations to ensure an appropriate study topic.
1.

The topic should be identified either as specified by the State Medicaid agency or
through data collection and analysis of comprehensive aspects of enrollee needs, care,
and services. Consider enrollee demographic characteristic and health risks, prevalence
of conditions, or the need for a specific service by enrollees.
A project topic also may be selected based on patterns of inappropriate utilization.
However, the project must be clearly focused on identifying and correcting deficiencies
in care or services that might have led to this pattern, such as inadequate access to
primary care, rather than on utilization or cost issues alone. The goal of the project
should be to improve processes and outcomes of health care. Therefore, it is acceptable
for a project to focus on patterns of over utilization that present a clear threat to health or
functional status.
Topics to be studied may also be selected on the basis of Medicaid enrollee input. To the
extent feasible, input from enrollees who are users of, or concerned with, specific focus
areas (e.g., mental health or substance abuse services) should be obtained from
individuals who use or are affected by these services.

2.

Study topics, over time, should address a broad spectrum of key aspects of enrollee care
and services including both clinical and nonclinical focus areas.
It is important that focused study topics represent the entire spectrum of clinical and
nonclinical areas associated with the MCO/PIHP, and also do not consistently eliminate
any particular subset of Medicaid enrollees; e.g., children with special health care needs.
Clinical focus areas should include, over time, prevention and care of acute and chronic
conditions, high-volume services, and high-risk services. High-volume services, as
opposed to a clinical condition, can include such services as labor and delivery, a
frequently performed surgical procedure, or different surgical or invasive procedures.
The study may also focus on high-risk procedures even if they are low in frequency; e.g.,
care received from specialized centers inside or outside of the organization=s network;
e.g., burn centers, transplant centers, cardiac surgery centers. The study may also assess
and improve the way in which the MCO/PIHP detects which of its members have special
health care needs and assess these members’ satisfaction with the care received from the
organization.
Finally, focused studies can address non-clinical areas. For example, focused studies that
address continuity or coordination of care can study the manner in which care is provided
when a patient receives care from multiple providers and across multiple episodes of
care. Such studies may be disease or condition-specific or may target continuity and

4

coordination across multiple conditions. Projects in other non-clinical areas can also
address, over time, appeals, grievance and complaints; or access to and availability of
services. Access and availability studies can focus on assessing and improving the
accessibility of specific services or services for specific conditions, including reducing
disparities between services to minorities and service to other members. Projects related
to the grievance and coverage determination process could aim either to improve the
processes themselves or to address underlying issues in care or services identified
through analysis of grievances or appeals.
Activity 2:

Define the Study Question(s)

Rationale. It is important to clearly state, in writing, the question(s) the study is designed to
answer. Stating the question(s) helps maintain the focus of the study and sets the framework for
data collection, analysis, and interpretation.
Potential Sources of Information to Help Form the Study Question:
-

State data relevant to the topic being studied
MCO/PIHP data relevant to the topic being studied
Relevant clinical literature

Methods of Implementation:
A study question(s) must be stated as clear, simple, answerable question(s). An example of a
vague study question is:
“Does the MCO/PIHP adequately address psychological problems in patients
recovering from myocardial infarction?”
In this example, it is not clear how “adequately address” will be assessed. Furthermore,
“psychological factors” is a very broad term. A clearer study question could be:
“Does doing ‘x’ reduce the proportion of patients with myocardial infarction who
develop severe emotional depression during hospitalization?”
Activity 3:

Select the Study Indicator(s)

Rationale. A study indicator is a quantitative or qualitative characteristic (variable) reflecting a
discrete event (e.g., an older adult has/has not received a flu shot in the last 12 months), or a
status (e.g., an enrollee’s blood pressure is/is not below a specified level) that is to be measured.
A study should have one or more quality indicators to assess performance. All indicators must
be objective, clearly and unambiguously defined, and based on current clinical knowledge or

5

health services research. In addition, all indicators must be capable of objectively measuring
either enrollee outcomes such as health or functional status, enrollee satisfaction, or valid proxies
of these outcomes.
Indicators can be few and simple, many and complex, or any combination thereof, depending on
the study question(s), the complexity of existing practice guidelines for a clinical condition, and
the availability of data and resources to gather the data.
Indicator criteria are the set of rules by which the data collector or reviewer determines whether
an indicator has been met. Pilot or field testing is helpful to the development of effective
indicator criteria. Such testing allows the opportunity to add criteria that might not have been
anticipated in the design phase. In addition, criteria are often refined over time, based on results
of previous studies. However, if criteria are changed significantly, the method for calculating an
indicator will not be consistent and performance on indicators will not be comparable over time.
It is important, therefore, for the indicator criteria to be developed as fully as possible during the
design and field testing of data collection instruments.
Potential Sources of Information to Help Select Study Indicators:
-

Clinical and non-clinical practice guidelines
Administrative data
Medical records

Methods of Implementation:
Address each of the following considerations to ensure an appropriate study indicator(s) is/are
identified.
1.

Each study should have objective, clearly defined, measurable indicators.
When indicators exist that are generally used within the public health community or the
managed care industry (such as NCQA’s Health Plan Employer Data and Information Set
(HEDIS) or the Foundation for Accountability’s (FACCT) measures) and these
indicators are applicable to the topic, use of those indicators is preferred. However,
indicators may be developed by the EQRO on the basis of current clinical practice
guidelines or clinical literature derived from health services research or findings of expert
or consensus panels
The following questions will assist in identifying meaningful indicators.
-

Are the indicator(s) related to identified health care guidelines pertinent to the study
question?

6

-

Do the indicators measure an important aspect of care that will make a difference to the
MCO’s/PIHP’s beneficiaries?

-

Are data available either through administrative data, medical records or other readily
accessible sources?

-

Will limitations on the ability to collect the data skew the results?

-

Do these indicators require explicit or implicit criteria? Consider the specificity of the
criteria used to determine compliance with an indicator. The greater number of people
involved in data collection and analysis, the greater the need for more explicit, or precise,
data collection and indicator criteria to obtain inter-reviewer reliability. The more
specific the criteria, the easier the data collection process will be, because staff will not
need extensive training. An example of an explicit criterion for an immunization study
is:
-

Documentation of refusal by parent to have a child immunized through nurses notes
and/or signed refusal by the parent in the medical record.
Implicit criteria may require a high degree of professional clinical judgement, and
therefore, may be time-consuming and expensive. An example of an implicit
criterion for an immunization study is:
-

Receipt of a childhood immunization is contraindicated.

Specific indicators do not always need to be established at the outset of a focused study.
There may be instances when a study may begin with more general collection and
analysis of baseline data on a topic, and then narrow its focus to more specific indicators
for measurement, intervention and reevaluation. The success of the focused study is
assessed in terms of the indicators ultimately selected.
2.

The indicators should measure changes in health status, functional status, or enrollee
satisfaction, or valid proxies of these outcomes.
The objective of a focused study should be to assess processes and outcomes. For
purposes of this protocol “outcomes” are defined as measures of patient health, functional
status or satisfaction following the receipt of care or services. Indicators selected for a
study of quality of care ideally should include at least some measures of change in health
or functional status or process of care “proxies” for these outcomes. Indicators may also
include measures of satisfaction.
It is recognized, however, that relatively few standardized performance measures actually
address outcomes. Even when outcome measures are available, their utility as quality
indicators may be limited because outcomes can be significantly influenced by factors

7

outside of the organization’s control, such as poverty, genetics, and the environment.
Because of this, quality indicators do not always need to be outcome measures. Process
measures are acceptable as long as it can be shown that there is strong clinical evidence
that the process being measured is meaningfully associated with outcomes. To the extent
possible, this determination should be based on published guidelines that support the
association and that cite evidence from randomized clinical trials, case control studies, or
cohort studies. Although published evidence is generally required, there may be certain
areas of practice for which empirical evidence of process/outcome linkage is limited. At
a minimum, it should be demonstrated that there is a consensus among relevant
practitioners with expertise in the defined area as to the importance of a given process.
Activity 4:

Use a Representative and Generalizable Study Population

Rationale. Once a topic has been selected, measurement and improvement efforts must be
system-wide; i.e., each study must represent the entire Medicaid enrolled population to which the
study indicators apply. Once that population is identified, the MCO/PIHP must decide whether
to review data for that entire population or use a sample of that population. Sampling is
acceptable as long as the samples are representative of the identified population (see Activity 5).
Potential Sources of Information to Promote Representativeness and Generalizeability of
the Study Population:
-

-

Data on the Medicaid enrolled population that enumerates the numbers of enrollees to
which the study topic and indicators apply. This would include demographic
information from MCO/PIHP enrollment files and MCO/PIHP utilization, diagnostic
and outcome information, such as services, procedures, admitting and encounter
diagnoses, adverse incidents (such as deaths, avoidable admissions, or readmissions),
and patterns of referrals or authorization requests.
Other data bases, as needed; e.g., pharmacy claims data to identify patients taking a
specific medication(s) during a specific enrollment period.

Methods of Implementation:
Address the following considerations to ensure that a representative and generalizable study
population is identified.
1.

Define the study’s “at risk” population.
-

All individuals to whom the identified study question(s) and indicators are
relevant must be defined.

-

Determine whether to include the entire study population or a sample in the
study. The decision may have been determined by the resources available to

8

-

analyze the data. If the State agency or MCO/PIHP is capable of collecting
and analyzing data through an automated data system, it might be possible to
study the whole population because many of the data collection and analysis
steps can be automated. If the data needs to be collected manually, sampling
may be more realistic.

-

Determine if the study population includes any requirements for the length of
the study population’s members enrollment in the MCO or PIHP. The
required length of time will vary depending on the study topic and study
indicators.

-

If the entire MCO/PIHP population is to be studied, the data collection
approach should capture all enrollees to which the study question applies.

If a sample is to be used, go to Activity 5. If the entire population is included in the study, skip
Activity 5 and go to Activity 6.
Activity 5:

Use Sound Sampling Techniques

Rationale. If a sample is to be used to select members of the study, proper sampling techniques
are necessary to provide valid and reliable (and therefore generalizable) information on the
quality of care provided. When conducting a study designed to estimate the rates at which certain
events occur, the sample size has a large impact on the level of statistical confidence in the study
estimates. Statistical confidence is a numerical statement of the probable degree of certainty or
accuracy of an estimate. In some situations, it expresses the probability that a difference could
be due to chance alone. In other applications, it expresses the probability of the accuracy of the
estimate. For example, a study may report that a disease is estimated to be present in 35% of the
population. This estimate might have a 95% level of confidence, plus or minus five percentage
points. This means that we are 95% sure that between 30-40 percent of the population has the
disease.
The true prevalence or incidence rate for the event in the population may not be known for the
first time a topic is studied. In such situations, the most prudent course of action is to assume
that a maximum sample size is needed to establish a statically valid baseline for the project
indicators.
Potential Sources of Information to Support Sampling:
-

-

Data on enrollee characteristics relevant to health risks or utilization of
clinical and non-clinical services, including age, sex, race/ethnicity/language
and functional status.
Utilization, diagnostic and outcome information, such as services, procedures,
admitting and encounter diagnoses, adverse incidents (such as deaths,

9

-

avoidable admissions, or readmissions), and patterns of referrals or
authorization requests.
Other information as needed, such as pharmacy claims data to identify
patients taking a defined number of a specific medication(s) during a specific
enrollment period.

Methods of Implementation:
Address the following factors to ensure appropriate sampling techniques are used.
1.

Determine the true (or estimated) frequency of occurrence of the event, the confidence
interval to be used, and the acceptable margin of error.

2.

Employ valid sampling techniques.
-

There are two basic categories of statistical sampling methods -- probability
sampling and nonprobability sampling.
Probability (or random) sampling methods leave selection of population units
totally to chance, and not to preference on the part of the individuals conducting
or otherwise participating in the study. Biases are removed in these methods.
There are several types of probability (or random) sampling that can be used:
-

In simple random sampling, all members of the study population have an
equal chance of being selected for the sample. Population members are
generally numbered, and random numbers generated by computer are used to
select units from the population.
Systematic random sampling - the basic principle is to select every nth unit in
a list. This can be used when a sampling frame is organized in a way that
does not bias the sample. Steps to organize and select a systematic sample
are:
1)

Construct a comprehensive sampling frame (e.g., list of all
beneficiaries).

2)

Divide the size of the sampling frame by the required sample size
to produce a sampling interval or skip interval (e.g., if there are
250 beneficiaries and a sample of 25 is needed, then divide 250/25
= 10).

3)

From a random number table select a random number between 1
and 10.

10

4)

Count down the list to get the Nth name (i.e., the # identified in
step 3).

5)

Skip down 10 names on the list and select a second name. Repeat
the process as many times as needed until the required sample size
has been reached.

-

Stratified random sampling is used when the target population consists of nonoverlapping sub-groups or strata. Typically this is used if the population is
homogeneous (same) within a strata and heterogeneous (different) between
strata. Stratified random sampling requires more information about the
population and also requires a larger overall sample size than simple random
sampling. Once strata are identified and selected, sampling must be
conducted within each strata using probability (or random) sampling.

-

Cluster sampling is used when a comprehensive sampling frame is NOT
available. Units in the population are gathered or classified into groups,
similar to stratified sampling. Unlike the stratified sampling method, the
groups must be heterogeneous within themselves with respect to the
characteristic being measured. This method requires prior knowledge about
the population. Once clusters are identified, a random sample of clusters are
selected.

Non-probability sampling methods are based on choice, rather than chance;
therefore some bias can be expected. There are several types of non-probability
sampling that can be used:
-

Judgment sampling involves constructing a sample based on including units in
the sample if they are thought (or judged) to be representative of the
population. By doing so, the sample is constructed to be a mini-population.

-

Convenience sampling uses units that are readily or conveniently available.
For example, if the objective were beneficiary opinions regarding a group
practice, patients in the office on any given day or during a specific month
could be interviewed.

-

Quota sampling ensures that units in the sample appear in the same proportion
as in the population. For instance, if a certain target population consisted of
55% female and 45% male, the quota sample would require a similar
female/male distribution.

11

Activity 6:

Reliably Collect Data

Rationale. Procedures used to collect data for a given study must ensure that the data collected
on the study indicators are valid and reliable. Validity is an indication of the accuracy of the
information obtained. Reliability is an indication of the repeatability or reproducibility of a
measurement. The strategy for developing a data collection plan should include:
-

clear identification of the data to be collected,
identification of the data,
specification of who will collect the data, and
identification of instruments used to collect the data.

When data are to be collected from automated data systems, development of specifications for
automated retrieval of the data is necessary. When data are obtained from visual inspection of
medical records or other primary source documents, several steps need to be taken to ensure the
data are consistently extracted and recorded:
1.

The key to successful manual data collection is in the selection of the data collection staff.
Appropriately qualified personnel, with conceptual and organizational skills, must be
used to abstract the data; however, the specific skills will vary with the nature of the data
being collected and the degree of professional judgment required. For example, when
data collection involves searching throughout the medical record to find and abstract
information or judging whether clinical criteria were met, experienced clinical staff, such
as registered nurses should collect the data. However, when the abstraction involves
verifying the presence of a diagnostic test report, trained medical assistants or medical
records clerks may be used.

2.

Clear guidelines for obtaining and recording data must be established, especially if
multiple reviewers are used to perform this activity. The qualifications of the data
collection staff should be determined before finalizing the data collection instrument. The
abstractor will need fewer clinical skills if the data elements within the data source are
more clearly defined. Developing a glossary of terms for each project should be a part of
the training of abstractors to ensure consistent interpretation among and between the
project staff.

3.

The number of data collection staff to be used for a given project affects the reliability of
the data. A smaller number of staff promotes inter-rater reliability; however, it may also
increase the amount of time it takes to complete this task. Intra-rater reliability (i.e.,
reproducibility of judgements by the same abstractor at a different time) should also be
considered.

Potential Sources of Data:
-

Administrative data; e.g., membership, enrollment, claims, encounters

12

-

Medical records
Tracking logs
Results of any provider interviews
Results of any beneficiary interviews and surveys

Methods of Implementation:
Address the following issues to ensure sound data collection procedures.
1.

The data to be collected should be clearly specified.
Accurate measurement depends on clearly defined data elements. Data elements must be
carefully specified with unambiguous definitions. When descriptive terms are used (e.g.,
“high,” “low,” “normal”), numerical definitions are established for each term. The units
of measure must also be specified (e.g., pounds, kilograms, etc.).

2.

The sources of data should be clearly specified.
Data sources vary considerably and depend upon the selected topic and indicators.
Similarly, the topic and indicators will reflect not just the clinical and research
considerations, but also the available data sources.

3.

A systematic method of collecting valid and reliable data that represents the entire
population to which the study’s indicators apply should be clearly defined.
The study may use automated or manual data collection methods depending on the
resources available. If an automated data collection system is utilized, the degree of
completeness of the data in the automated system is always a concern. For example, for:
-

Inpatient data: The data system should capture all inpatient admissions.

-

Primary care data: Data for all encounters should be available.

-

Specialty care data: Data for all encounters should be available.

-

Ancillary services data: Encounter or utilization data should be available for
all services provided.

The study’s design and methodology should include an estimation of the degree
of completeness of the automated data available for the study indicators. 2

2

The accuracy of automated data is also a concern, but validation of this is beyond the scope of this

protocol.

13

Manual data collection may be the only feasible option for many topics selected. The
beneficiary medical record is the most frequently used data source. Other manual
systems which might contain sources of information include clinical tracking logs,
registries, complaint logs, and manual claims.
When using manual data collection, the design of the focused study should reflect that:

4.

-

Study staff and personnel have appropriate clinical knowledge and skills,
including good conceptual, organization, and documentation skills.

-

Data collection instruments provide for reliable and accurate data collection
over the time period to be studied.

-

If manual data collection is to be performed, the data collection instrument(s)
should be clear and promote inter-rater reliability. An important part of
designing data collection instruments is developing instructions or guidelines
for data collection staff. Instrument design is particularly important when
staff not involved in the study design perform data collection. Instructions
should be clearly and succinctly written and should provide an overview of
the study, specific instructions on how to complete each section of the form
and general guidance on how to handle situations not covered by the
instructions.

-

When assessing non-clinical services such as health care access or cultural
competency or care coordination, a study may utilize information on how the
MCO/PIHP is structured and operates.

The study design should specify a data analysis plan that reflects the following
considerations:
-

Whether qualitative or quantitative data, or both, will be collected.
Qualitative data describes characteristics or attributes by which persons or things
can be classified; for example, sex, race, poverty level, or the presence or absence
of a specific disease. Calculation of proportions and calculation of rates are the
two most common qualitative measures.
Quantitative data are concerned with numerical variables such as height, weight
and blood levels. The methods by which the data are analyzed and presented will
vary by type of data. Quantitative data require, at a minimum, simple descriptive
statistics such as measures of central tendency (i.e., mean, median or mode) and
measure of variability (i.e., range or standard deviation).

-

Whether the data will be collected on the entire population or a sample.

14

-

Whether the measurements obtained from the data collection activity will be
compared to the results of previous or similar studies. If so, the data analysis plan
should have considered evaluating the comparability of the studies and identified the
appropriate statistical tests to be used to compare studies.

-

Whether the study will be compared to the performance of an individual MCO/PIHP,
a number of MCOs/PIHPs, or different provider sites. Comparing the performance of
multiple entities involves greater statistical design and analytical considerations than
those required for a study of a single entity, such as a MCO/PIHP.

Activity 7:

Analyze Data and Interpret Study Results.

Rationale. Data analysis begins with examining the MCO’s/PIHP’s performance on the selected
clinical or non-clinical indicators. The examination should be initiated using statistical analysis
techniques defined in the data analysis plan.
Potential Sources of Data and Information:
-

Baseline project indicator measurements
Industry benchmarks
Analytic reports of focused study results by the MCO/PIHP

Methods of Implementation:
Address the following considerations to ensure that data analysis and interpretations are
appropriate and valid.
1.

The analysis of the findings should be conducted according to the data analysis plan.

2.

The results and findings should present numerical study data in a way that provides
accurate, clear, and easily understood information.

3.

Following the data analysis plan, the analysis should identify:

4.

-

measurable results of the MCO/PIHP performance in comparison to
prospectively identified goals or benchmarks

-

factors that threaten the internal or external validity of the findings.

-

statistical significance of any apparent differences between units of
comparisons.

-

factors that influence the comparability of the data.

The analysis of the study data should include an interpretation of the extent to which the
focused study was successful and what follow-up activities are planned as a result.

15

Interpretation and analysis of the study data should be based on continuous improvement
philosophies and reflect an understanding that most problems result from failures of
administrative or delivery system processes, not failures of individuals within the system.
Interpreting the data should involve developing a hypotheses about the causes of lessthan-optimal performance and collecting data to validate the hypotheses.
END OF PROTOCOL

16

ATTACHMENT A

CONDUCTING FOCUSED STUDIES OF HEALTH CARE
WORKSHEET
Use this or a similar worksheet as a guide while designing and conducting focused studies.
Document the completion of each step. Refer to the protocol for detailed information on each
area.

Demographic Information
MCO/PIHP Name or ID:
Study Leader Name:
Telephone Number:
Name of Focused Study:
Date of Study Period:

____ Staff Model
____ Network
____ Direct IPA
____ IPA Organization

/

/

to

/

/

Type of Delivery System (check all that are applicable)
____ MCO
_____ Number of Medicaid Enrollees in MCO or PIHP
____ PIHP
_____ Number of Medicare Enrollees in MCO or PIHP
_____ Number of Medicaid Enrollees in Study
_____ Total Number of MCO or PIHP Enrollees in Study

Number of MCO/PIHP primary care physicians ____________
Number of MCO/PIHP specialty physicians _______________
Number of physicians in study ____________

17

ATTACHMENT A

Component/Standard Number
Activity 1.

Comments

Date
Comp.

SELECT THE STUDY TOPIC(S)

1.1. Study topic is selected through data
collection and analysis of comprehensive
aspects of enrollee needs, care and services.

1.2. The topic(s), over time, address a broad
spectrum of key aspects of enrollee care and
services.
1.3. The topics, over time, include all enrolled
populations: i.e., do no exclude certain
enrollees such as those with special health
care needs.

Activity 2.

DEFINE THE STUDY QUESTION(S)

2.1. The study question(s) is/are clearly stated in
writing.
Activity 3.

SELECT STUDY INDICATOR(S)

3.1. The study has objective, clearly defined,
measurable indicators.
3.2. The indicators measure changes in health
status, functional status, or enrollee
satisfaction, or valid proxies of these
outcomes.
Activity 4.

USE A REPRESENTATIVE AND GENERALIZABLE STUDY POPULATION

4.1. The at-risk population is defined.
4.2. If the study includes the entire population,
the data collection approach captures all
enrollees to whom the study question
applies

18

ATTACHMENT A

Activity 5.

USE SOUND SAMPLING TECHNIQUES

5.1. The sampling technique considers and
specifies the true frequency of occurrence,
the confidence interval and the margin of
error.
5.2. A sufficient number of enrollees are
sampled.
5.3. Valid sampling techniques are used.

Activity 6.

RELIABLY COLLECT DATA

6.1. The data to be collected are clearly
specified.
6.2. The sources of data are clearly specified.
6.3. The methods of collecting data are clearly
defined.
6.4. The data collection instruments provide for
consistent, accurate data collection.
6.5. The study design specifies a data analysis
plan.
6.6. Qualified staff and personnel are used to
collect the data.

19

ATTACHMENT A

Activity 7.

ANALYZE DATA AND INTERPRET STUDY RESULTS

7.1. Analysis of findings are conducted
according to the data analysis plan.
7.2. Results and findings present numerical data
in a way that provides accurate, clear and
easily understood information.
7.3. The analysis identifies measurable results in
comparison to identified goals and
benchmarks, factors that threaten internal
and external validity, statistical significance
of apparent differences between units of
comparisons, and factors that influence the
comparability of data.
7.4. The analysis includes an interpretation of
the extent to which the study was successful
and follow-up activities.

20

ATTACHMENT A
Record any additional comments pertinent to the design and/or conduct of the study:

21

ATTACHMENT B

ORIGIN OF THE PROTOCOL
This protocol was one of nine protocols developed during 1998-2001 from standards and
guidelines used in the public and private sectors during this time. This protocol was developed
from the following documents:
-

Quality Improvement System for Managed Care (QISMC)
QISMC was an initiative of CMS that set forth standards and guidelines pertaining to
health care quality for Medicaid and Medicare health plans (MCOs, PIHPs, and
Medicare+Choice plans). These standards and guidelines, in part, address MCO and PIHP
quality assessment and improvement projects.

-

Health Care Quality Improvement Studies in Managed Care Settings: A Guide for State
Medicaid Agencies (National Committee for Quality Assurance (NCQA))
Produced under a contract from CMS, this guidebook identifies key concepts related to the
conduct of QI studies and details widely accepted principles of research design and
statistical analysis necessary for designing, implementing and assessing QI studies.

-

A Health Care Quality Improvement System for the Medicaid Managed Care, A Guide for
States (Health Care Financing Administrations (HCFA))
CMS’s 1993 guide for health care QI provides a framework for building QI systems
within State Medicaid managed care initiatives. This document included guidelines
addressing quality assessment and improvement studies and related activities of MCOs
and PIHPs. This document was the result of the Quality Assurance Reform Initiative
(QARI).

-

Framework for Improving Performance, From Principles to Practice (Joint Commission
on Accreditation of Healthcare Organizations (JCAHO)
This publication describes the Joint Commission’s theory-based, practical methodology
for continuously improving the core work and resulting outcomes of any health care
organization. In this document, JCAHO defines the key characteristics and essential
behaviors of any health care organization striving to achieve high quality patient care.

-

1990-2000 Standards for Health Care Networks (SHCN) (JCAHO)
The JCAHO 1990-2000 SHCN provides a standards-based evaluation process to assist the
MCO in measuring, assessing and improving its network’s performance. It also helps the
MCO focus on conducting performance improvement efforts in a multi disciplinary,
system-wide manner. The 1990-2000 SHCN integrates information about the Joint
Commission’s health care network accreditation process.

22

ATTACHMENT B
-

NCQA 1997, 1998, and 1999 Standards for Accreditation of Managed Care
Organizations and NCQA 1999 Standards for Accreditation of Managed Behavioral
Healthcare Organizations (MBHO)
These documents include administrative policies and procedures for NCQA’s MCO and
MBHO accreditation programs, the 1997, 1998, and 1999 standards, and rationale
statements for the standards.

-

Peer Review Organizations (PRO) 4th and 5th Scope of Work (SOW) (CMS)
The 4th and 5th SOW documents outlined the requirements for PROs to adhere to while
conducting health care quality and improvement activities for Medicare beneficiaries.

An in-depth comparison of these documents was performed to identify the activities and features
common to these protocols, and features unique to individual protocols, while acknowledging the
different purposes of the documents. The QISMC, JCAHO and NCQA standards are written as
guides for MCOs/PIHPs to follow in developing, conducting and evaluating their quality
improvement studies. They can also be used by States or their agents (e.g., EQROs) to assess
compliance with State mandated guidelines and/or to facilitate overall plan-to-plan comparisons.
QARI was written with States as the intended audience to help them and their agents (e.g.,
EQROs) assure compliance with regulations and Medicaid program requirements, and promote
consistency in the manner in which MCOs and PIHPs carry out activities related to focused
studies.
The analysis revealed that in spite of their different purposes, all the documents identify several
common characteristics of effective focused studies. These include:
Selection of Topics: All of the reference documents address the need for focused studies to clearly
specify the topic to be addressed. They all acknowledge both clinical (e.g., specific disease or
condition such as pregnancy or asthma) and non-clinical (e.g., availability, timeliness and
accessibility of care) health service delivery issues as appropriate topics for health care QI
initiatives.
Means of Identifying Topics: Continuous data collection and analysis is stressed throughout all
documents as a means of identifying appropriate topics. It is stated that topics should be
systematically selected and prioritized to achieve the greatest practical benefit for enrollees. A
minimal set of criteria is suggested for selecting appropriate topics, including: the prevalence of a
condition among, or need for a service by, the MCO’s/PIHP’s enrollees; enrollee demographic
characteristics and health risks; the likelihood that the study topic will result in improved health
status among the enrollees; and the interest of consumers in the aspect of care or services to be
addressed.
Scope of study topics: The QISMC standards specify that performance improvement projects
should address the breadth of the MCO’s or PIHP’s services, such as whether they include

23

ATTACHMENT B
physical health and mental/substance abuse health services. They also identify specific clinical
and non-clinical focus areas that are applicable to all enrollees. The QISMC standards also
specify that the scope of the health plans’ improvement efforts are to include all enrollees.
Stating the Study Question(s): The HCQIS Guide discusses the importance of “stating the study
question” after a study topic is identified. It asserts that stating a study question helps a project
team avoid becoming sidetracked by data that is not central to the issue under study. For example,
once a focused study has identified childhood immunizations as a study topic: it might specify a
number of different study questions:
-

Have all children received all scheduled doses of one vaccine in particular?

-

Have all children of all ages received all recommended vaccines appropriate
for their age?

-

Have all children of a particular age (e.g., at the age of one, two, six or other
years) received all age-appropriate immunizations?

Alternatively, more detailed information may be desired so it may be necessary to specify
the study questions as:
-

What proportion of Medicaid enrollees who have reached two years of age
have received:
- All four recommended doses of DPT vaccine?
- All three recommended doses of the Polio vaccine?
- One recommended dose of the MMR vaccine?
- At least one dose of Hib in the second year of life?

Further specificity of additional study questions may be desired to provide information in
QI efforts, such as:
-

In what percent of cases of lack of immunization were children not immunized
for one of the following reasons?
- Refusal by a parent or guardian.
- Medical contraindications.
- Member non-complaint with the recommended immunization regimen.

Incorporating the process of documenting a study question(s) into the project design can help
ensure a systematic method of identifying appropriate indicators and data to be collected. In this

24

ATTACHMENT B
protocol we have included “defining the study question(s)” as a key step in designing and
implementing a Focused study.
Use of Quality Indicators: All reference documents address the need to specify well-defined
indicators to be monitored and evaluated throughout the study. It is emphasized that quality
indicators do not always need to be outcome measures. Process measures are also appropriate,
especially when there is strong clinical evidence that the process being measured has a
meaningful association with outcomes. There are various ways to obtain appropriate indicators,
such as using those dictated from outside sources (such as the State or CMS) or by an MCO/PIHP
developing them internally on the basis of clinical literature or findings of expert panels.
In addition to these features found uniformly in all reference documents, other significant aspects
of focused studies were identified by one or more of the reference documents. These include:
Significant improvement: NCQA’s document, “Health Care Quality Improvement Studies in
Managed Care Settings”, states that, “When presenting statistical results of any study, it is
important to fully disclose. . .the statistical significance of the estimates produced, as well as the
statistical significance of any apparent differences between units of comparison.” Building on
this, CMS’s QISMC document called for specific amounts of measurable improvement to be
demonstrated by the health plan. QISMC defines “demonstrable” improvement as either: 1)
benchmarks established by CMS (for national Medicare projects) or State agencies (for statewide
Medicaid QI projects) or by the health plans for individual (organizational) projects; or 2) a 10%
reduction in adverse outcomes. This protocol does not call for a specific level of statistical
achievement to be achieved but, consistent with the NCQA document, calls for disclosure and
review of the statistical significance of any measurable performance of a focused study.
Phase-in or time frame requirements: QISMC delineates specific time frame requirements for
MCOs/PIHPs to reach certain phases in a QI cycle. For example:
-

By the end of the first year, an MCO/PIHP should have initiated at least two
quality improvement projects addressing two different focus areas;

-

By the end of the second review year, at least two additional projects
addressing two different focus areas should be initiated.

-

By the end of the first year after the 2 year phase-in period, and each
subsequent year, at least two projects are to achieve demonstrable
improvement in two of the focus areas.

Evaluation Tools: NCQA’s HCQIS guidebook includes study planning and summary worksheets
to be used in the evaluation of an MCO’s/PIHP’s focused study. This feature provides a helpful
method for recording data during the evaluation process and promotes the collection of consistent

25

ATTACHMENT B
information by all evaluators. This protocol contains an example of a worksheet (Attachment A)
that can be used by EQROs when conducting focused studies.
Scoring system: NCQA accreditation provides a numerical scoring system to measure
performance against standards and to promote consistency in the process used to evaluate MCOs.
Although the scores do not dictate the final decision with respect to compliance with standards,
they do serve as a guide for NCQA evaluators to recommend non-compliance. This scoring
system also includes an opportunity for the MCO/PIHP to comment on the reviewer’s scores
before a final decision is rendered. It also promotes continuous improvement practices by
securing “customer” input into a final product (i.e., evaluation decisions). This protocol does not
include a scoring system.
END OF DOCUMENT

26