1-ehrdpraomb_a_b

Contract No.:
500-2005-00025I (0006)
MPR Reference No.: 6479-150

Evaluation of the
Electronic Health
Records Demonstration
(EHRD) and the
Medicare Care
Management
Performance (MCMP)
Demonstration:
Supporting Statement for
Paperwork Reduction Act
Submission
Final
April 10, 2009
Martha Kovac
Nancy Duda
Mindy Hu
Jennifer Schore

Submitted to:
US Department of Health and Human Services
Centers for Medicare & Medicaid Services
Office of Research, Development, and Information
C3-23-04 Central Bldg.
7500 Security Blvd.
Baltimore, MD 21244-1850
Project Officer:
Lorraine Johnson

Submitted by:
Mathematica Policy Research, Inc.
P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director:
Jennifer Schore

CONTENTS

Chapter

Page

A.BACKGROUND .........................................................................................................................1
1.
2.
3.

RATIONALE FOR THE MCMP AND EHR DEMONSTRATIONS ....................1
MCMP AND EHR DEMONSTRATION DESIGNS..............................................3
MCMP AND EHRD EVALUATION DESIGNS ...................................................7

B.JUSTIFICATION .......................................................................................................................15
1.
2.
3.
4.
5.
6.
7.
8.

NEED AND LEGAL BASIS .................................................................................15
INFORMATION USERS ......................................................................................16
USE OF INFORMATION TECHNOLOGY.........................................................16
DUPLICATION OF EFFORT...............................................................................16
SMALL BUSINESSES .........................................................................................17
LESS-FREQUENT COLLECTION ......................................................................18
SPECIAL CIRCUMSTANCES .............................................................................19
FEDERAL REGISTER/OUTSIDE CONSULTATION .......................................19
a.

9.
10.
11.
12.
13.
14.
15.
16.

Public Comment and Responses ....................................................................19

PAYMENTS/GIFTS TO RESPONDENTS ..........................................................20
CONFIDENTIALITY............................................................................................21
SENSITIVE QUESTIONS ....................................................................................22
BURDEN ESTIMATES (HOURS AND WAGES) ..............................................23
CAPITAL COSTS .................................................................................................26
COST TO FEDERAL GOVERNMENT ...............................................................26
CHANGES TO BURDEN .....................................................................................28
PUBLICATION/TABULATION DATES ............................................................28
a.
b.
c.
d.
e.
f.
g.

Cost Neutrality Monitoring Report ................................................................29
Interim and Final Evaluation Reports ............................................................29
Report to Congress .........................................................................................30
Implementation Reports .................................................................................30
Final Site Visit Report ...................................................................................31
Cost Neutrality Monitoring Reports ..............................................................32
Interim Evaluation Summary and Final Evaluation Reports .........................32

17. EXPIRATION DATE ............................................................................................33
18. CERTIFICATION STATEMENT.........................................................................33

iii

CONTENTS (continued)
Chapter

Page

C.SUPPORTING STATEMENT- PART B COLLECTION OF
INFORMATION EMPLOYING STATISTICAL METHODS ....................................................34
1.

RESPONDENT UNIVERSE AND SAMPLING METHODS .............................34
a.
b.

2.
3.
4.
5.

Respondent Universe .....................................................................................34
Sampling Methods .........................................................................................36

PROCEDURES FOR THE COLLECTION OF INFORMATION .......................41
METHODS TO MAXIMIZE RESPONSE RATES ..............................................42
TESTS OF PROCEDURES OR METHODS ........................................................49
PEOPLE INVOLVED IN DESIGN ......................................................................50

REFERENCES ..............................................................................................................................52

iv

APPENDICES

Appendix

Page

A

LEGISLATION FOR THE MEDICARE CARE MANAGEMENT
PERFORMANCE DEMONSTRATION .............................................................A.3

B

LEGISLATION FOR THE ELECTRONIC HEALTH RECORDS
DEMONSTRATION ........................................................................................... B.3

C

FEDERAL REGISTER NOTICE ........................................................................ C.3

D

ELECTRONIC HEALTH RECORDS DEMONSTRATION (EHRD)
ADVANCE LETTERS ........................................................................................ D.3

E

ELECTRONIC HEALTH RECORDS DEMONSTRATION (EHRD)
FACT SHEETS .................................................................................................... E.3

F

ELECTRONIC HEALTH RECORDS DEMONSTRATION (EHRD)
OFFICE SYSTEMS SURVEY ............................................................................ F.3

G

ELECTRONIC HEALTH RECORDS DEMONSTRATION (EHRD)
VALIDATION FORM ........................................................................................G.3

H

MEDICARE CARE MANAGEMENT PERFORMANCE (MCMP)
DEMONSTRATION: ADVANCE LETTERS ................................................... H.3

I

MEDICARE CARE MANAGEMENT PERFORMANCE (MCMP)
DEMONSTRATION: FACT SHEETS ................................................................ I.3

J

MEDICARE CARE MANAGEMENT PERFORMANCE (MCMP)
DEMONSTRATION OFFICE SYSTEMS SURVEY ......................................... J.3

K

ELECTRONIC HEALTH RECORDS DEMONSTRATION (EHRD)
PRACTICE DISCUSSION GUIDES ..................................................................K.3

L

ELECTRONIC HEALTH RECORDS DEMONSTRATION (EHRD)
ADVANCE LETTER FOR DISCUSSIONS WITH PRACTICES ..................... L.3

v

TABLES

Table

Page

A.1

ELECTRONIC HEALTH RECORDS DEMONSTRATION SITES
AND COMMUNITY PARTNERS BY SITE .........................................................5

A.2

MEASURES COLLECTED ON THE MCMP OFFICE SYSTEMS
SURVEY ..................................................................................................................8

A.3

MEASURES COLLECTED ON THE EHRD OFFICE SYSTEMS
SURVEY ................................................................................................................12

A.4

EHRD QUALITATIVE DISCUSSION TOPICS BY TYPE OF
RESPONDENT ......................................................................................................13

B.1

CONSULTANTS ...................................................................................................20

B.2

EHRD OSS ANNUAL RESPONSE BURDEN ....................................................24

B.3

EHRD OSS VALIDATION OF SURVEY RESPONSES ANNUAL
RESPONSE BURDEN ..........................................................................................25

B.4

EHRD DISCUSSIONS WITH PRACTICE STAFF AND COMMUNITY
PARTNERS ANNUAL RESPONSE BURDEN ...................................................27

B.5

MCMP DEMONSTRATION DELIVERY SCHEDULE OF REPORTS .............28

B.6

EHRD DELIVERY SCHEDULE OF REPORTS .................................................31

C.1

DISTRIBUTION OF ELIGIBLE PRACTICES, BY SIZE, IN MCMP
DEMONSTRATION STATES ..............................................................................35

C.2

MCMP DEMONSTRATION NUMBER OF ENROLLED PRACTICES BY
SITE .......................................................................................................................37

C.3

EXPECTED OSS COMPLETES BY DEMONSTRATION AND YEAR ...........40

C.4

EHRD PRACTICE DISCUSSIONS BY YEAR AND ROUND ..........................41

vii

TABLES (continued)
Table
C.5

Page
OSS DATA COLLECTION ACTIVITY BY WEEK ...........................................43

viii

FIGURES

Figure

Page

I.1

TIMELINE FOR THE EHR DEMONSTRATION AND EVALUATION
AND FOR THE MCMP OSS ADMINISTRATION ............................................14

ix

x

A. BACKGROUND

CMS contracted with Mathematica Policy Research, Inc. (MPR) to conduct an evaluation of
two pay for performance demonstration projects (the congressionally mandated Medicare Care
Management Performance (MCMP) demonstration and the new Electronic Health Record
Demonstration (EHRD) which is a high priority of the Administration and Secretary). Both
demonstrations are testing different methods of payment for improving the quality and efficiency
of care to Medicare fee-for-service beneficiaries with chronic conditions and for encouraging the
use of health information technology (HIT) to improve care delivery. The demonstrations are a
component of the Administration’s broad HIT strategy to ensure that most Americans have
access to secure, interoperable health records by 2014. They align with the goals of the Office of
the National Coordinator for Health Information Technology (ONC) to inform and interconnect
health care providers, personalize care, and improve population health through EHR systems
(Thompson and Brailer 2004).
We are requesting approval of the Office System Survey (OSS) that will be used in the
MCMP and EHR demonstrations, and approval of the in-person and telephone discussion guides
for use with community partners and physician practices in the EHR demonstration.
1.

Rationale for the MCMP and EHR Demonstrations
The MCMP demonstration was authorized under Section 649 of the Medicare Prescription

Drug, Improvement, and Modernization Act of 2003 (MMA) (see Appendix A). This is a three
year pay for performance demonstration with physicians to promote the adoption and use of HIT
to improve the quality of care for eligible chronically ill Medicare beneficiaries. MCMP targets
small to medium sized primary care practices with up to 10 physicians. Practices must provide
care to at least 50 Medicare beneficiaries. Approximately 650 practices in Arkansas, California,

1

Massachusetts and Utah are currently participating in this demonstration which started July 1,
2007. The demonstration is expected to end June 30, 2010. Physicians will receive payments for
meeting or exceeding performance standards for quality of care. They will also receive an
additional incentive payment for electronic submission of performance measures via their
electronic health record (EHR) system. These payments are in addition to their normal payments
for providing service to Medicare beneficiaries. The OSS will be used to assess progress of
physician practices in implementation and use of EHRs and related HIT functionalities.
The EHR demonstration is authorized under Section 402 of the Medicare Waiver Authority
(see Appendix B). The goal of this five year pay for performance demonstration is to foster the
implementation and adoption of EHRs and HIT in order to improve the quality of care provided
by physician practices. The EHRD expands upon the MCMP Demonstration and will test
whether performance-based financial incentives (1) increase physician practices’ adoption and
use of electronic health records (EHRs), and (2) improve the quality of care that practices deliver
to chronically ill patients. The EHRD targets small to medium sized primary care practices with
up to 20 physicians. Practices must provide care to at least 50 Medicare beneficiaries. The
demonstration began operations June 1, 2009, and is expected to end May 31, 2014.
Approximately 800 practices will be enrolled in the demonstration across four sites. Practices will
be randomly assigned to a treatment and control group. The OSS will be used to assess progress
of physician practices in implementation and use of EHRs and related HIT functionalities, and to
determine incentive payments for treatment practices. In-person and telephone discussions with
community partners and physician practices will be used to learn about practices’ experiences
and strategies in adopting and using EHRs, as well as the factors that help or hinder their efforts.

2

2.

MCMP and EHR Demonstration Designs

MCMP Demonstration Design
The MCMP demonstration will target practices serving at least 50 traditional fee-for-service
Medicare beneficiaries with selected chronic conditions for whom they provide primary care.
Under this demonstration, physicians practicing primary care 1 in solo or small- to medium-sized
group practices (practices with 10 or fewer physicians, although there may be exceptions) will be
eligible to earn incentive payments for (1) reporting quality measures for congestive heart failure
(CHF), coronary artery disease (CAD), diabetes, and the provision of preventive health services
during a baseline (predemonstration) period; (2) achieving specified standards on clinical
performance measures during the three-year demonstration period; and (3) submitting clinical
quality measures to CMS electronically using an EMR system that meets industry standards
specified by the Certification Commission for Healthcare Information Technology (CCHIT).
The legislation authorizes up to four demonstration sites to include both urban and rural
areas. 2 The states of Arkansas, California, Massachusetts, and Utah were chosen as the four sites.
The Quality Improvement Organizations (QIOs) in these four states recruited demonstration
practices on relationships built through CMS’s Doctor’s Office Quality - Information
Technology (DOQ-IT) project. Demonstration practices represent many organizational
structures, and serve at least 50 Medicare beneficiaries. Recruitment of demonstration practices
began in January 2007.
1

The following physician specialties will be eligible to participate in the MCMP demonstration if they provide
primary care: general practice, allergy/immunology, cardiology, family practice, gastroenterology, internal
medicine, pulmonary disease, geriatric medicine, osteopathic medicine, nephrology, infectious disease,
endocrinology, multispecialty clinic or group practice, hematology, hematology/oncology, preventive medicine,
rheumatology, and medical oncology.
2

In addition, the statute requires that one site be “in a state with a medical school with a Department of
Geriatrics that manages rural outreach sites and is capable of managing patients with multiple chronic conditions,
one of which is dementia.”

3

The demonstration practices will be eligible to receive up to three incentive payments. First,
demonstration practices will receive an incentive of $20 per beneficiary per category (up to
$1,000 per physician to a maximum of $5,000 per practice) for reporting baseline clinical quality
measures. The payment will not be contingent on the practice’s score on any of these measures.
Second, for each of the three demonstration years, based on the clinical measures data that
the practices report, CMS will calculate a composite score for each chronic condition (as well as
the preventive measures) and compare it against performance thresholds. Physicians will be
eligible for payments of up to $70 per beneficiary for meeting standards related to a specific
chronic condition. Beneficiaries who have more than one condition will be counted in each of the
relevant groups. For preventive services, physicians will be eligible for a payment of up to $25
per beneficiary with any chronic condition. Physicians will be eligible to earn up to $10,000 per
year for performance on all clinical measures. The maximum annual payment to any single
practice will be $50,000, regardless of the number of physicians in the practice.
Third, practices with a CCHIT certified EMR system that can extract and submit
performance data to CMS electronically will be eligible to increase the incentive payment by up
to 25 percent, or $2,500 per physician (up to $12,500 per practice) per year during the
demonstration period for electronic submission. Thus, practices could receive up to $192,500
over the three years of the demonstration (including the baseline period).
EHRD Design
The EHRD targets small to medium-sized practices (20 or fewer physicians, though there
may be exceptions) providing primary care 3 to fee-for-service Medicare beneficiaries with

3

The following physician specialties will be eligible to participate in the EHRD demonstration if they provide
primary care: general practice, allergy/immunology, cardiology, family practice, gastroenterology, internal
medicine, pulmonary disease, geriatric medicine, osteopathic medicine, nephrology, infectious disease,

4

congestive heart failure, coronary artery disease, diabetes, or other chronic diseases. CMS plans
to recruit 800 small to medium-sized practices in four selected sites (about 200 per site). CMS
has identified and recruited community partners in the four sites to help with practice outreach,
education, and recruitment. Community partners represent community stakeholders and have ties
to primary care physicians, but they are not required to have a specific type of organizational
entity or structure. (Table A.1 presents the four sites and the affiliated community partners.)
The demonstration will begin operations on June 1, 2009, and will end on May 31, 2014. In
spring 2009, prior to the start of demonstration operations, practices within each site will be
randomly allocated to a treatment or a control group. MPR will design the randomization
approach and will conduct the randomization.
TABLE A.1
EHRD DEMONSTRATION SITES AND COMMUNITY PARTNERS
Demonstration Sites

Community Partners

Louisiana

Louisiana Health Care Quality Forum

Maryland/DC

MedChi & Maryland Health Care Commission

Pennsylvania—Pittsburgh and surrounding counties

Pittsburgh Regional Health Initiative

South Dakota and some border counties in Iowa,
Minnesota, and North Dakota

South Dakota Department of Health/South
Dakota E-Health Collaborative

The evaluation will use one or more tax identification numbers (TINs) to identify
demonstration practices. Physicians will be linked to practices using TINs and individual
provider identification number (PIN) and/or National Provider Identification (NPI). Medicare
beneficiaries who live in a demonstration site and who are treated by primary care providers (or

(continued)
endocrinology, multispecialty clinic or group practice, hematology, hematology/oncology, preventive medicine,
rheumatology, and medical oncology.

5

those medical subspecialties likely to provide primary care for the targeted conditions) and who
are covered under traditional fee-for-service Medicare for both Part A and Part B will potentially
be linked to these practices. 4
Under the demonstration, treatment practices will be eligible to receive up to three types of
payments, which will be distributed by ARC. The first is a payment (up to $5,000 per physician
to a maximum of $25,000 per practice) for using an EHR, called the systems payment, in all five
years of the demonstration, beginning in year 1. The systems payment will be based on the
practice’s use of a minimum set of functions in an EHR system certified by the Certification
Commission for Healthcare Information Technology (CCHIT), as measured by the practice’s
responses to the OSS. Additional payment will be provided for use of more-sophisticated EHR
functions, such as identifying medication interactions. Practices that have not adopted or
implemented minimal use of the EHR system by the end of the first year will not receive a
payment, but they may remain in the demonstration.
In year 2, practices have the opportunity to receive the systems payment and a reporting
payment (up to $3,000 per physician to a maximum of $15,000 per practice), for reporting on
specific clinical quality measures. Practices that have not adopted or implemented minimal use
of the EHR system by the end of the second year will be removed from the demonstration. In
years 3 to 5, practices will have the opportunity to receive the systems payment and a quality
payment (up to $10,000 per physician to a maximum of $50,000 per practice), based on
performance on specific clinical quality measures, such as blood pressure management and lipid
measurement for beneficiaries with diabetes. These financial payments will be in addition to the
normal fee-for-service Medicare payment that practices receive for services delivered.
4

Beneficiaries for whom Medicare is not the primary source of insurance coverage or whose care is managed
by a hospice program will be excluded from the demonstration.

6

Physicians could receive up to $58,000 and practices up to $290,000 over the five years of the
demonstration (Wilkin et al. 2007).
3.

MCMP and EHRD Evaluation Designs

MCMP Evaluation Design
The main goal of the evaluation is to provide CMS and AHRQ with valid estimates of the
incremental effect, or impact, of providing performance-based financial incentives on the quality
of care, continuity of care, use of Medicare-covered services, and Medicare costs of the
chronically ill Medicare beneficiaries served by demonstration practices. To provide this
information, the evaluation must generate rigorous quantitative estimates of the intervention’s
impacts.
The impact analysis for the evaluation will use a matched comparison group design (that is,
it will use a comparison group or quasi-experimental design). The impact analysis will use a
difference-in-differences approach to estimate impacts. With this approach, changes in quality
measures and other outcomes of practices in the demonstration states and comparison states will
be compared before and after the start of the demonstration. The unit of analysis will be the
practice, which also is the unit of intervention.
Data for impact analysis will be collected from four sources: (1) an Office Systems Survey,
(2) a physician survey, (3) a beneficiary survey, and (4) Medicare claims and eligibility data.
This request for OMB clearance includes the administration, during fall 2009, of the second and
final round of the OSS for the MCMP Demonstration evaluation. The first round was conducted
in 2007 by another CMS contractor as part of the Quality Improvement Organization/Eighth
Scope of Work and was therefore exempt from the OMB clearance process. The OSS will
measure practices’ adoption and use of EHRs and its specific functions, including prescribing

7

medications, ordering laboratory tests and other procedures, and care management and
coordination (see Table A.2 for a list of key survey topics).
TABLE A.2
MEASURES COLLECTED ON THE MCMP OFFICE SYSTEMS SURVEY
Office Practices and Work Flows
Proportion of patient visits where paper charts are pulled; providers dictate notes into a tape recorder or
phone; use computerized system to manage office work flows
Use of Electronic Health Records
Availability of EHR system
Whether EHR system is CCHIT certified
Proportion of paper records that have been transitioned to HER
Use of EHR system to perform functions (for example, documenting office visits, e-prescribing,
polypharmacy, or issuing laboratory orders or patient reminders)
Use of stand-alone systems for e-prescribing or patient registries
Practice Characteristics
Practice contact information (name, address, telephone number)
Affiliations with IPAs, PHOs, other medical groups
Participation in other quality improvement or pay-for-performance initiatives
Provider Characteristics
Provider name and identification number
Credentials
Specialty
Languages spoken (other than English)
Medicare billing number (provider identification number or PIN)

EHRD Evaluation Design
The main goal of the EHRD evaluation is to provide CMS with valid estimates of the
incremental effect, or impact, of providing performance-based financial incentives on the
adoption and use of EHR, quality of care, continuity of care, use of Medicare-covered services,
and Medicare costs of chronically ill Medicare beneficiaries served by demonstration practices.
To provide this information, the evaluation must generate rigorous quantitative estimates of the
intervention’s impacts.

8

The impact analysis will compare regression-adjusted outcome measures for the treatment
and control groups in order to test hypotheses concerning the impact of the combined financial
incentives (that is, the systems, reporting, and quality performance payments). The evaluation
also includes a descriptive quantitative analysis of the chart-based quality measures, which are
available only for the treatment group.
The impact analysis will assess impacts on outcomes measured at the practice, physician,
and beneficiary levels, whereas the intervention is implemented only at the practice level.
Because the practice—not the beneficiaries—receives the demonstration payments, it is the unit
of analysis. When sample sizes permit, the impact analysis will use hierarchical linear models
that nest beneficiaries within practices to assess the impact of the demonstration on outcomes
(for example, quality of care, service use, and Medicare expenditures) that are measured at the
beneficiary level.
Because physician practice regulations, practice styles, practice settings, adoption of EHRs,
and pay-for-performance penetration are likely to differ across sites and may render data pooling
infeasible, the impact analysis will estimate impacts separately for each demonstration site.
However, for certain analyses, such as those based on survey data, the analysis will explore
pooling data across sites. Where sample sizes permit, we will estimate impacts for subgroups
defined by practice features (such as size or patient mix) and by beneficiary characteristics.
Data for the impact analysis will be collected from five sources: (1) the demonstrations’
practice application form, (2) the OSS, (3) a beneficiary survey, (4) a physician survey, and (5)
Medicare claims and eligibility data.
A second goal of the evaluation is to provide CMS with a detailed understanding of
practices’ experiences and strategies in adopting and using EHRs, and the factors that help or
hinder their efforts. To provide this information, the evaluation will use qualitative techniques

9

and descriptive quantitative methods to gather and analyze information gathered from practice
staff and community partners. Data for the implementation analysis will come from in-person
and telephone discussions with practice staff and community partners, and practice responses to
the OSS.
A third goal of the evaluation is to provide CMS with a synthesis of quantitative and
qualitative findings to answer three questions: (1) What are the overall impacts of the
demonstration? (2) What is the relationship between the level of the systems and quality
performance payments and changes in quality-of-care indicators and use of EHRs? and (3) What
is the relationship between the quality-of-care indicators and health outcomes? Data sources for
the overall impact analysis include site-specific findings from all four sites and outcome
measures from both the implementation and the impact analyses. The analysis will rely on an
implementation synthesis to summarize how the intervention was implemented across the four
sites, an exploratory analysis to compare the characteristics of successful and unsuccessful sites,
a confirmatory analysis to test for differences in impacts between groups of sites, and a synthesis
of findings on the types of beneficiaries for whom the intervention was most effective.
Data sources for the analysis of the relationship between system levels and quality factors
include practice-level data for treatment group practices in each site (systems and quality
performance payments and scores, chart-based quality measures and scores, measures of use of
EHRs, and health outcomes derived from claims data); when appropriate, data across all sites
will be pooled to maximize the available sample size. This analysis will use linear regression
models to estimate the association between changes in quality of care and use of EHRs and the
level of systems and quality performance payments. We will use similar models to estimate the
association between quality-of-care indicators and health outcomes.

10

This request for OMB clearance relates only to the design and conduct of the EHRD’s OSS
and the in-person and telephone discussions with practice staff and community partners, since
these activities are expected to begin in 2010. 5 The OSS will measure practices’ use of EHRs
and their specific functions, including prescribing medications, ordering laboratory tests and
other procedures, and care management and coordination (see Table A.3 for a list of key survey
topics). Treatment group practices will be required to complete an annual OSS for each of the
five demonstration years. To minimize respondent burden, control group practices will be asked
to complete the survey at the end of demonstration years 2 and 5 only. In addition, MPR will
conduct a yearly validation with 25 percent of the responding treatment practices to validate their
responses to the OSS.
Discussions with staff of participating practices will gather their perspectives on the
demonstration and incentives; their experience in adopting and implementing HIT, including
barriers and facilitators; their use of HIT functions; and their view of the effect of new HIT on
care management, quality measures and improvement activities, and practice operations. These
discussions will take place during years 1 and 5 of the demonstration’s operations. Discussions
with staff of practices that have withdrawn from the demonstration will provide information on
their reasons for participation and withdrawal, their participation and attitudes toward pay-forperformance programs, and factors that would have encouraged them to remain in the
demonstration. Discussions with community partners will provide information on the recruitment
and operational experiences of the practices, which will inform the interpretation of the

5

We will submit a separate request for OMB clearance for the beneficiary and physician surveys, since we
expect these to be fielded about 36 months after practice enrollment in the demonstration, starting roughly in June
2012.

11

interviews with the practices (see Table A.4 for a list of key discussion topics by respondent
type).
TABLE A.3
MEASURES COLLECTED ON THE EHRD OFFICE SYSTEMS SURVEY
Use of Electronic Health Records
Availability of EHR system
Whether EHR system is CCHIT certified
Number of providers in the practice who use the EHR system
Proportion of paper records that have been transitioned to EHR
Use of EHR system to perform functions (for example, documenting office visits, e-prescribing,
polypharmacy, or issuing laboratory orders or patient reminders)
Use of stand-alone systems for e-prescribing or patient registries
Practice Characteristics
Practice contact information (name, address, telephone number)
Affiliations with IPAs, PHOs, other medical groups
Participation in other quality improvement or pay-for-performance initiatives
Provider Characteristics
Provider name and identification number
Credentials
Specialty
Languages spoken (other than English)
Medicare billing number (provider identification number or PIN)

The overall timeline for the demonstration and evaluation activities is presented in Figure
I.1.

12

TABLE A.4
EHRD QUALITATIVE DISCUSSION TOPICS BY TYPE OF RESPONDENT

Practices
Administrative Staff
Practice demographics
Practice perspective on the demonstration and early response
Adaptation of practice operations as HIT is implemented
Facilitators and barriers to adopting and implementing HIT
Other incentives, reporting programs, and HIT initiatives
Use of HIT for care management
Plans for change

Physicians
Demonstration participation and operational response
HIT experience and effect on practice change
Care management views and experience
Quality measures and improvement activities

Medical Director
Physicians’ use of HIT functions
Physicians and other clinical staff attitudes toward HIT
Other issues related to HIT adoption, such as the effect on malpractice insurance premiums
Changes in job responsibilities or patient interface
Critical factors for success and closing

Nurse or Other Staff Member Performing Care Management
Effect of new HIT, or changes in use, on job responsibilities
Adoption of care management
Greater use of data to refine the care process
Enhanced practice orientation to quality and safety

Senior Administrative Personnel (CEO, CFO, Marketing Director)
Demonstration’s fit with practice goals
Effects of HIT on the practice
Expectations/thinking regarding incentive payments
Market factors

Community Partners
Experience recruiting practices to the demonstration
Practice needs for the demonstration to be successful
Plans for working with and facilitating assistance to practices
Perceptions of practices’ progress under the demonstration
Other (non-EHRD) HIT activities in the site

13

FIGURE I. 1
TIMELINE FOR THE EHR DEMONSTRATION AND EVALUATION
AND FOR THE MCMP OSS ADMINISTRATION

Mar. 2008

Evaluation contract awarded

June 2008

Demonstration s ites selecteda

June 2009

Demonstration starts

Sep. 2009

MCMP OSS

Mar. 2010

Site vis its (round 1)
OSS (treatment practices)

Mar. 2011

OSS (treatment and control practices)

Mar. 2012

Withdrawn practices interviews
OSS (treatment practices)

Mar. 2013

OSS (treatment practices)

Mar. 2014

Site vis its (round 2)
OSS (treatment and control practices)

May 2014

Demonstration ends

Feb 2016

Evaluation ends

Note: Medicare claims data will be collected annually for treatment and control groups.

a

In spring 2009, practices will be randomly allocated within each site to a treatment or control group.

14

B. JUSTIFICATION

1.

Need and Legal Basis
The MCMP demonstration was authorized under Section 649 of the Medicare Prescription

Drug, Improvement, and Modernization Act of 2003 (MMA). This three year, pay for
performance demonstration with physicians will promote the adoption and use of HIT to
improve the quality of care for eligible chronically ill fee for service Medicare beneficiaries. The
MMA authorized up to four demonstration sites to include urban and rural areas; CMS chose
Arkansas, California, Massachusetts, and Utah. The legislation mandated an independent
evaluation of the MCMP Demonstration. The evaluation must include an assessment of the
impact of providing pay-for-performance financial incentives on quality of care, care
coordination, and continuity of care, thereby reducing Medicare expenditures and improving
health outcomes. This evaluation requires an OSS to determine adoption and use of HIT in
physician practices.
The EHRD is authorized under Section 402 of the Medicare waiver authority. The EHRD
expands upon the MCMP Demonstration and will test whether systems and performance-based
financial incentives (1) increase physician practices’ adoption and use of EHRs, and (2) improve
the quality of care that practices deliver to chronically ill patients with fee-for-service Medicare
coverage. This demonstration requires an OSS to determine HIT use and financial incentives. In
addition, in-person visits and telephone calls with practices and community partners are
necessary to learn how HIT and care management are implemented within practices and across
sites.
Both demonstrations are a component of the Administration’s broad HIT strategy to ensure
that most Americans have access to secure, interoperable health records by 2014. The

15

demonstrations align with the goals of the Office of the National Coordinator for Health
Information Technology (ONC) to inform and interconnect health care providers, personalize
care, and improve population health through EHR systems (Thompson and Brailer 2004).
2.

Information Users
Information for the evaluation of the MCMP Demonstration will be collected and analyzed

by MPR, under Contract Number 500-00-0033, Task Order 05 with CMS, titled “Evaluation of
the Medicare Care Management Performance Demonstration.” Findings from the impact analysis
will be included in the Report to Congress (due within 12 months of the conclusion of the
demonstration) and other internal reports to CMS.
Information for the evaluation of EHRD will be collected and analyzed by MPR, under
Contract Number HHSM-500-2005-00025I (0006) with CMS, titled “Evaluation of the
Electronic Health Records Demonstration.” Findings from the implementation analysis, impact
analysis, and synthesis of findings will be included in the internal reports to CMS.
3.

Use of Information Technology
The MCMP and EHR demonstrations will use a web-based survey instrument (OSS) as the

primary method of data collection with practices, supplemented by paper-and-pencil
questionnaires for those practices that either request a paper version or do not have internet
access early on in the demonstration. Multiple attempts will be made to encourage practices to
complete the web-based survey. While we expect that most practices will complete the web
survey, a small number of practices may prefer to complete a paper version. MCMP OSS data
collection will take place in 2009, one year prior to the end of the demonstration. The EHRD
OSS data collection with treatment group practices will take place at the end of each
demonstration year; data collection with control group practices will be at the end of

16

demonstration years 2 and 5. All participating demonstration practices will receive an advance
letter instructing them how to log onto the website to complete the OSS online. The letter will
also provide a toll-free number to call if a practice has a question or prefers to complete a paperand-pencil questionnaire. A toll-free help desk and general email address will be established to
assist practices in completing the online survey. We expect that 90 percent of responding
practices will complete the web-based version of the OSS, and 10 percent will complete a paperand-pencil version.
In-person and telephone discussions with EHRD practices and community partners will take
place in 2010 and 2014. Discussions with practices that have withdrawn from the demonstration
will occur during spring 2012. All the discussions will be conducted using semi-structured,
paper-and-pencil discussion guides. Information gathered from the practice discussions will be
entered into Atlas.ti software (a commercial tool for analyzing qualitative data) to help identify
themes and illustrative examples.
4.

Duplication of Effort
These information collections do not duplicate any other effort, and the information cannot

be obtained from any other source.
5.

Small Businesses
For the MCMP Demonstration, small to medium-sized practices (with 10 or fewer

physicians) were targeted. For the EHRD, small to medium-sized practices (with 20 or fewer
physicians) will be targeted. Participating in the OSS for the MCMP and EHR demonstrations
will impose minimal burden, as the surveys are designed to be completed in 29 minutes. The
web-survey format permits practice staff to complete the survey at their convenience.
Furthermore, only one survey is needed from each practice at each data collection point.

17

Respondent signatures are not required for either of the demonstrations’ OSSs. However, we will
collect from respondents, once they have completed the survey, a data attestation that verifies the
accuracy of the information they provided, along with their name and title.
Two rounds of discussions (in-person or telephone) will be conducted with up to 24
practices and 4 community partners during demonstration years 1 and 5. Each round of contacts
will include site-level discussions with 4 treatment group practices, up to 2 control group
practices, and the community partner. At each of the 24 practices, discussions with three key
staff members will be scheduled at the convenience of the practice, during physicians’ off-hours
if necessary. Most discussions with practice staff and community partners will last 30 minutes,
and none will be scheduled to exceed one hour. In addition, discussions with up to 6 practices
that have withdrawn from the demonstration will be conducted during spring 2012. Each
discussion will last 30 minutes.
6.

Less-Frequent Collection
Collecting OSS data at regular intervals is critical for tracking changes in the adoption and

use of HIT, and for conducting a credible evaluation of both the MCMP and EHR
demonstrations. Conducting a second round of the OSS for the MCMP demonstration is
necessary to measure changes in the use of EHR and HIT over time. For the EHRD, the OSS
must be administered annually to treatment group practices in order to determine the amount of
the demonstration incentive payment. Yearly payments to treatment practices would be
impossible with less-frequent administration. Administering the survey to the control group at
years 2 and 5 is necessary to measure changes in the use of EHRs over time in the absence of
demonstration incentives. Less-frequent collections of OSS data would limit CMS’s
understanding of the impact of the MCMP and EHR demonstrations.

18

For the EHRD, conducting two rounds of discussions with practice staff and community
partners is necessary to provide details of how practices are evolving in their use of technology
to support care quality. Conducting discussions with practice staff and community partners at
only one point would make it impossible to describe these changes over time. Conducting
discussions with practices that have withdrawn from the demonstration is necessary to capture
the characteristics and reasons for withdrawal.
7.

Special Circumstances
There are no special circumstances related to the proposed data collection.

8.

Federal Register/Outside Consultation
The notice required by 5 CFR 1320.8 (d), was published in the Federal Register on January

23, 2009. A copy of the notice is in Appendix C.
a.

Public Comment and Responses
Public comments were received at the conclusion of the first 60-day period and written

responses have been submitted.
Outside consultation for the design of the studies and surveys was received from a variety of
experts (see Table B.1 for a list of consultants).
The OSS instruments developed for the second (and final) round of administration for the
MCMP and for the EHR demonstration evaluations drew heavily upon the OSS instrument that
was designed and administered in 2007 under the MCMP evaluation. In fact, the current MCMP
OSS instrument is nearly identical; the only difference is the removal of a short set of questions
that are no longer applicable (those pertaining to experience and satisfaction with Quality
Improvement Organizations). The MCMP OSS questionnaire was pretested with less than nine
respondents.

19

TABLE B.1
CONSULTANTS
Individual
James Sorace

Affiliation/Agency/Division
Within the Department of Health and Human Services
Assistant Secretary for Planning and Evaluation

(202) 205-8678

Jody Blatt

Centers for Medicare & Medicaid Services

(410) 786-6921

Debbie Vanhoven

Centers for Medicare & Medicaid Services

(410) 786-6625

Karen Bell

Office of the National Coordinator for Health Information
Technology

(202) 690-7151

David Hunt

Office of the National Coordinator for Health Information
Technology

(202) 690-7151

John Wilkin

Outside the Department of Health and Human Services
Actuarial Research Corporation

Telephone Number

(703) 941-7400

For the EHRD, we used the 2007 MCMP OSS to identify questions that were asked
successfully of a similar population. We added questions to capture EHR functions and use in
greater detail. The EHRD OSS questionnaire was pretested with less than nine respondents.
9.

Payments/Gifts to Respondents
No incentive payment was offered for the initial round of the MCMP OSS in 2007.

Therefore, we will not offer an incentive to practices for participating in the second round of the
OSS for the MCMP demonstration.
For the EHRD, treatment group practices must participate in the OSS to receive the systems
payments (described in Part A of the OMB submission package). For this reason, treatment
practices will have a strong motivation to participate in the OSS and an incentive payment will
not be needed to ensure a high response rate. On the other hand, control group practices receive
no demonstration payment for adoption and use of an EHR, so they will have no clear incentive

20

to complete the OSS. We therefore plan to offer a $50 incentive to control group practices to
ensure a comparably high response rate to the survey. Without an incentive for the control group,
there is a risk of obtaining a lower response rate for control group practices, which could bias the
study results.
No incentive payment will be offered for participation in the EHRD discussions with
practice staff and community partners.
10. Confidentiality
MPR will take several steps to assure respondents that the information they provide will be
treated as confidential and used for research purposes only. Advance letters to practices will
inform respondents that data collected from the OSS or practice discussions will be aggregated in
reports and that practice-level data will not be reported. MPR will restrict access to the OSS web
instrument to protect the confidentiality of respondents and the preloaded practice-level
information the instrument contains. Each practice will be assigned a unique ID and password
that will be included in the advance letter addressed to the person who completed the
demonstration application form (for the initial round of the survey) or the prior OSS (for
subsequent rounds).
The OSS web instrument will be hosted on MPR’s web servers. Data will be processed and
stored on MPR’s password-protected local area network (LAN). MPR protects its LAN with
several security mechanisms available through the network operating system: Novell Netware
5.1, IntraNetware, and a firewall from Cisco Systems. Novell Netware 5.1 is compliant with the
C2/E2 Red Book security specifications. IntraNetware is certified at the National Computer
Security Center’s Trusted Network Interpretation Class C2 level at the network level. All LAN
servers containing confidential information are located in a controlled-access area, which is also
protected from unauthorized external electronic access by a firewall from Cisco Systems. This

21

firewall is located between the T1 line, which connects to the Internet, and the rest of MPR’s
network. Access to confidential information stored on LAN directories is restricted to authorized
project staff by means of ID and password. In addition, network servers containing confidential
information are kept in a locked area. All staff working with extremely sensitive data are
required to change their password at least every 90 days. In addition, LAN access privileges for
staff leaving the project are revoked within 24 hours.
Completed paper-and-pencil questionnaires are sensitive documents, since they contain both
personal identifiers (name, phone number, address) and survey data. MPR staff will create a
detailed plan for tracking and protecting the OSS paper-and-pencil instruments through the data
collection process (that is, quality control, data entry, and coding). Once the paper questionnaires
are received at MPR, personal identification information will be removed and separated from the
respondent’s survey data as soon as possible. A unique identification number will be used to link
or connect the personal identifiers to the respondent’s survey data. The linking methodology will
be secured to prevent unauthorized linkage of the survey data and the personal identifiers.
Finally, MPR staff assigned to work on the project all sign confidentiality pledges as a term
of employment. The confidentiality pledge requires that staff maintain the confidentiality of all
information collected.
11. Sensitive Questions
The MCMP and EHR OSS instruments include questions about practices’ use of EHRs and
the EHRs’ specific functions, including prescribing medications, ordering laboratory tests and
other procedures, and care management and coordination. These questions are not considered
sensitive. Many of the questions were adapted without modification from the previous round of
the OSS administered for the MCMP demonstration.

22

For the EHRD, the questions asked during discussions with practice staff cover their
perspectives on the demonstration and incentives; experience in adopting and implementing an
EHR, including barriers and facilitators; use of HIT functions; and the effect of new HIT on care
management, quality measures and improvement activities, and practice operations. These
questions are not considered sensitive. The questions asked during discussions with practices that
have withdrawn from the demonstration cover the reasons for enrollment and withdrawal from
the demonstration, participation and attitudes toward pay-for-performance programs, and factors
that could have encouraged them to remain in the demonstration. These questions are not
considered sensitive.
12. Burden Estimates (Hours and Wages)
Table B.2 presents estimates of respondent burden for completing the OSS for the MCMP
and EHR demonstration evaluations. It shows the expected number of respondents, the hours per
response, and the annualized hour and cost burden for each year that OSS data are collected. The
OSS for the MCMP Demonstration will be administered in fall 2009. Practices enrolled in the
EHRD will be surveyed beginning in Spring 2010, and treatment group practices will be
surveyed annually thereafter; control group practices will be surveyed in the Spring of years 2
and 5 of the demonstration. Hourly estimates for the OSS are based on pretest interviews
completed with less than nine practices. Interview completion times ranged from 24 to 35
minutes for the MCMP OSS and 25 to 37 minutes for the EHRD OSS, with an average length of
29 minutes for each survey instrument. The cost per practice was computed using an estimated
annual salary of $40,000 for practice managers and 2,080 annual work hours as follows:
$40,000/2,080*0.48 = $9 per response.

23

TABLE B.2
EHRD OSS ANNUAL RESPONSE BURDEN

Number of
Respondents*

Frequency
of
Response

Hours Per
Response

Annual
Hour
Burden

Cost Per
Response

Annual
Cost
Burden

980

1

0.48

470

$9

$8,820

EHRD (2010)

400

1

0.48

192

$9

$3,600

EHRD (2011)

680

1

0.48

384

$9

$7,200

EHRD (2012)

380

1

0.48

192

$9

$3,600

EHRD (2013)

380

1

0.48

192

$9

$3,600

EHRD (2014)

660

1

0.48

384

$9

$7,200

Study (Year)
MCMP
Demonstration
(2009)

*Assumes that 5 percent of treatment group practices will withdraw at the end of demonstration year 2, and
that 70 percent of control group practices will complete a survey in demonstration year 2 and 5.

Table B.3 presents estimates of respondent burden for completing a validation of OSS
responses for the EHRD. It shows the expected number of respondents, the hours per response,
and the annualized hour and cost burden for each year that OSS data are collected. The
validation will take place each year the OSS is conducted, with 25 percent of the treatment
practices that completed a survey, and will occur after the OSS data have been collected. Hourly
estimates for the OSS are based on pretest interviews completed with less than nine practices.
Interview completion times ranged from 60 to 105 minutes, with an average length of 83
minutes. The cost per practice was computed using an estimated annual salary of $40,000 for
practice managers and 2,080 annual work hours as follows: $40,000/2,080*1.38 = $26 per
response.

24

TABLE B.3
EHRD OSS VALIDATION OF SURVEY RESPONSES ANNUAL RESPONSE BURDEN

Study (Year)

Number of
Respondents

Frequency of
Response

Hours Per
Response

Annual
Hour
Burden

Cost Per
Response

Annual Cost
Burden

EHRD (2010)

100

1

1.38

138

$26

$2,600

EHRD (2011)

100

1

1.38

138

$26

$2,600

EHRD (2012)

95

1

1.38

131

$26

$2,470

EHRD (2013)

95

1

1.38

131

$26

$2,470

EHRD (2014)

95

1

1.38

131

$26

$2,470

Discussions with EHRD practice staff and community partners will be conducted during
years 1 and 5 of the evaluation. Within each site, discussions will be held with 4 treatment
practices, up to 2 control practices, and a community partner, for a total of 24 practices and 4
community partners. Discussions will be conducted at two points in time and both rounds will be
conducted in person. In each round, discussions with each practice will last two to three hours,
depending on practice size and staffing structure. MPR staff will meet with three key staff
members at each practice for discussions using semi-structured guides, which vary in length
from 30 minutes to one hour each depending upon the staff member’s involvement in the
implementation of HIT. An additional 45-minute group discussion may be scheduled for larger
practices that have additional administrative personnel who could inform the project. Discussions
with community partners will be 60 minutes long.
Telephone discussions will be held with up to 6 practices that have withdrawn from the
demonstration. These 30-minute discussions will occur in year 1 of the evaluation.
Table B.4 presents estimates of respondent burden for completing the discussions with
EHRD practices and community partners. It shows the expected number of respondents, hours

25

per response, and the annualized hour and cost burden for each year that discussions are
conducted. Hourly estimates for the discussions are based on mock interviews completed with
MPR staff. Interview completion times ranged from 30 to 60 minutes, with an average length of
45 minutes. The cost per discussion was computed using an estimated annual salary of $40,000
for practice managers and community partners; $160,000 for physicians, medical directors and
CFOs; and $47,000 for nurses, with 2,080 annual work hours as follows: practice managers and
community partners: $40,000/2,080*1.0 = $19 per response; physicians, medical directors and
CFOs: $160,000/2,080*0.75 = $58 per response; and nurses: $47,000/2,080*0.5 = $11 per
response.
13. Capital Costs
There are no direct costs to respondents other than their time to participate in the study.
14. Cost to Federal Government
The total current value for the MCMP evaluation contract is $2,299,876 over seven years.
The estimated annualized cost to the government for conducting the OSS is $141,369. This
figure is based on the contractor’s costs for collecting and tabulating the survey data, including
labor; other direct costs for computer, telephone, postage, reproduction, fax, printing, and survey
facilities; and indirect costs for fringe benefits, general and administrative costs, and fees.
The total current value for the EHRD contract is $5,225,643 over eight years. The estimated
annualized cost to the government for conducting the OSS is $158,457, and for conducting the
practice discussions is $125,486. These estimates are based on the contractor’s costs for
collecting and tabulating survey and practice contact data, including labor and travel; other direct
costs for computer, telephone, postage, reproduction, fax, printing, and survey facilities; and
indirect costs for fringe benefits, general and administrative costs, and fees.

26

TABLE B.4
EHRD DISCUSSIONS WITH PRACTICE STAFF, COMMUNITY PARTNERS
AND PRACTICES THAT WITHDREW FROM THE DEMONSTRATION
ANNUAL RESPONSE BURDEN
Round 1 Discussions (Evaluation year 1)

Type of Respondent
Discussions with
Practice staff**

Number of
Respondents

Frequency
Hours per
of Response Response

Annual
Hour
Burden

Annual Cost
per Response

Cost
Burden

72

1

0.75

54

$29

$2,088

Discussions with
Community Partners

4

1

1.0

4

$19

$76

Discussions with
Withdrawn Practices

6

1

0.5

3

$19

$114

Round 2 Discussions (Evaluation year 5)

Hours per
Response

Annual
Hour
Burden

0.75

54

$29

$2,088

Discussions with
Community Partners
4
1
1.0
_______________________________________________

4

$19

$76

Type of Respondent
Discussions with
Practice staff**

Number of
Respondents
72

Frequency
of
Response
1

Annual Cost
per Response

Cost
Burden

*Total number of respondents over both years
--Practice staff: 144
--Community Partners: 8
--Practices that withdrew: 6
**Number of practices per round: 24; Number of respondents per practice: 3

15. Changes to Burden
Data collection for EHRD is new; therefore, there are no changes to burden. For the MCMP
demonstration, there is a slight reduction in burden for the second round of the OSS, a result of
the removal from the 2007 version of a short series of questions that pertained to Quality
Improvement Organizations and are no longer relevant.

27

16. Publication/Tabulation Dates
MCMP Demonstration Evaluation Reports
The MCMP Demonstration evaluation will produce several reports, including an interim and
a final evaluation report that synthesize findings across states and analytic components. The
evaluation reports will be adapted to develop the Report to Congress. Table B.5 summarizes the
delivery schedule. A summary of each report follows.
TABLE B.5
MCMP DEMONSTRATION DELIVERY SCHEDULE OF REPORTS
Due Date
Report

Project Month

a

Calendar Month

First Interim Evaluation Report

19

January 2009

Cost Neutrality Monitoring Report

25

July 2009

Second Interim Evaluation Report

30

December 2009

Report to Congress
(Third Interim Evaluation Report)

43

January 2011

Final Evaluation Report

53

November 2011

a

Refers to the number of months after the start of the demonstration (July 1, 2007).

a.

Cost Neutrality Monitoring Report
OMB has requested that CMS monitor cost neutrality over the first 18 months of the

demonstration. This analysis will require comparing regression estimates of the demonstration’s
effects on Medicare savings to the incentive payments made to demonstration practices.
Assuming the data for this analysis are available by month 22 (that is, 22 months after the
demonstration begins), we will deliver a draft report to CMS in month 25 (July 2009).

28

b. Interim and Final Evaluation Reports
Three interim evaluation reports (drafts due 19, 30, and 43 months after the start of the
demonstration) and a final evaluation report (draft due 53 months after the start of the
demonstration) will be prepared for CMS, all of which will synthesize those findings available at
different times during the demonstration.
The first interim evaluation report, due January 2009 (19 months after the start of the
demonstration), will provide qualitative descriptions of practice changes made in response to the
intervention, including changes to the processes associated with the adoption of HIT and how it
is used.
The second interim evaluation report, due in December 2009 (30 months after the start of
the demonstration), will focus on impact estimates for the first year of program operations.
Although the report will compare impacts on use of Medicare-covered services and costs across
practices and states, we will not attempt to draw inferences from them at this stage of the
evaluation. In addition, the report will summarize findings from telephone discussions with
highly successful practices and with those that withdrew, if any, in year 2 of demonstration
operations.
The third interim evaluation report, due to CMS in January 2011 (43 months after the start
of the demonstration), will focus on impact estimates for the second year of program operations.
The report will also include findings on the impacts of pay-for-performance on physicianbeneficiary interactions (that is, access to care, care coordination, and satisfaction with care)
from the beneficiary survey.
The final evaluation report, due in November 2011 (53 months after the start of the
demonstration), will provide final impact estimates from claims data from the third, and final,
year of demonstration operations. In addition, the report will present impact estimates from the

29

physician survey on processes associated with the adoption of HIT to improve quality of care. It
will also incorporate our synthesis analysis, including data from the last wave of the OSS.
c.

Report to Congress
MPR will produce one Report to Congress based on the independent evaluation. The report

is due in January 2011, about six months after the end of demonstration operations. This report
will analyze implementation experiences and findings of the MCMP Demonstration across the
four states.
EHRD Evaluation Reports
The EHRD evaluation will produce several reports, including implementation reports, final
site visit reports, and interim and final evaluation reports. Table B.6 summarizes the delivery
schedule. A description of each report follows.
d. Implementation Report
The results from the analysis based on the first round of site visits will be presented in the
implementation report, due to CMS in June 2010 (13 months after the start of the
demonstration). The implementation analysis report will provide an overview of practice
characteristics and demonstration implementation in the first year of operation. It will rely on
data from reports for the first round of site visits to practices, and will include data from practice
applications for the demonstration and the OSS, as available. 6

6 Most treatment group practices will not yet have completed the first annual OSS, and OSS data for the
control group practices will not be available for the implementation analysis report, since they will not be collected
from control group practices until the second year of the demonstration.

30

TABLE B.6
EHRD DELIVERY SCHEDULE OF REPORTS
Due Date
Report

Project Month

a

Calendar Month

Implementation analysis report

13

June 2010

Interim evaluation report

40

September 2012

Evaluation summary report

58

March 2014

Final site visit report

70

March 2015

Evaluation final report

79

December 2015

Final cost monitoring report

81

February 2016

a

Refers to the number of months after the start of the demonstration (June 1, 2009).

e.

Final Site Visit Report
Results from the second round of contacts to practices will be reported in the final site visit

report, due to CMS in March 2015 (70 months after the start of the demonstration). The final site
visit report will draw implementation-related conclusions, related primarily to EHR use and care
management activities. 7 It will synthesize information from the implementation report, the 8 site
visit reports, and descriptive analyses of OSS data. The analysis of OSS data will examine
changes in use of EHR functions over time, while the qualitative analysis of the site contact
information will offer insights about how practices responded to demonstration incentives.

7 The final site visit report may be delayed if the second round of site visits is postponed. Contact with
practices may be postponed to fully capture the potential effects of the quality performance payments, which will
first be received by practices in the third year of the demonstration.

31

f.

Cost Neutrality Monitoring Reports
The cost-monitoring reports, due annually between December 2010 and 2014 (19 to 67

months after the start of the demonstration), will analyze whether the demonstration is generating
cost savings and will include annual and cumulative estimates of the impact of the demonstration
payments on use of Medicare-covered services and expenditures. The final cost-monitoring
report, due in February 2016 (81 months after the start of the demonstration), will summarize
findings for the demonstration’s impacts on Medicare expenditures; it will draw from the
evaluation final report described below. The cost-monitoring reports will rely primarily on data
from Medicare claims and on the levels of the demonstration payments made to treatment group
practices for each year of demonstration operations.
g.

Interim Evaluation Summary and Final Evaluation Reports
The interim evaluation report, due in September 2012 (40 months after the start of the

demonstration), will provide descriptions of practice changes made after three years of
intervention and will discuss the initial perceived effects of the demonstration on EHR use,
quality of care, and costs from the implementation analysis. It will summarize findings from the
first round of site visits to practices, as reported in the implementation report, and telephone
interviews with practices that withdrew, if any. The report will also include findings from the
first cost reports, which will draw on Medicare claims data and the incentive payments.
The evaluation summary report, due in March 2014 (58 months after the start of the
demonstration), will provide qualitative descriptions of practice changes made after 4.5 years of
intervention and will discuss impact estimates on quality of care, use of EHR, use of Medicarecovered services, and costs across sites. The report will synthesize findings from the practice
contacts, interviews with representatives of withdrawn practices, and OSS data (including year 2
data for treatment and control group practices). Finally, the report will include impact analyses of

32

OSS-based systems scores, use of Medicare-covered services and expenditures, and claims-based
quality measures, as available.
The final evaluation report, due in December 2015 (79 months after the start of the
demonstration), will provide final impact estimates on the use of Medicare-covered services and
expenditures, claims-based quality measures, survey-based measures, and OSS-based systems
scores, using data from the final year of demonstration operations.8 The report will also include a
synthesis analysis of the overall effects of the systems and performance financial incentives on
quality measures, and EHR use. It will rely on findings from the final site visit reports, the fifth
cost-monitoring report, the last year of the OSS, the beneficiary and physician survey analysis
reports, and the interim and summary evaluation reports.
17. Expiration Date
The OMB expiration date will be displayed on all materials sent to practices, including the
advance letter and a paper version of the OSS questionnaire.
18. Certification Statement
Data collection efforts for the MCMP and EHRD OSS and for the EHRD discussions with
practice staff and community partners will conform to all provisions of the Paperwork Reduction
Act.

8 Completion of the final evaluation report may require an extension to allow for final collection of quality
performance data and claims run out.

33