Parts_A-B

PART A
SUPPORTING STATEMENT

1

MEDICAID EMERGENCY PSYCHIATRIC DEMONSTRATION EVALUATION
A. Background
Since the inception of Medicaid, inpatient care provided to adults ages 21 to 64 in
institutions for mental disease (IMDs) has been excluded from federal matching funds. The
Emergency Medical Treatment and Active Labor Act (EMTALA), however, requires IMDs that
participate in Medicare to provide treatment for psychiatric emergency medical conditions
(EMCs), even for Medicaid patients for whose services they cannot be reimbursed. Section 2707
of the Patient Protection and Affordable Care Act (ACA) of 2011 (P.L. 111-148; Attachment A)
directs the Secretary of Health and Human Services to conduct and evaluate a demonstration
project to determine the impact of providing payment under Medicaid for inpatient services
provided by private IMDs to individuals with emergency psychiatric conditions between the ages
of 21 and 64. This project, the Medicaid Emergency Psychiatric Demonstration (MEPD), and its
evaluation are being implemented by the Centers for Medicare & Medicaid Services (CMS). On
May 10, 2011, CMS received Paperwork Reduction Act (PRA) approval to collect application
information from states interested in participating in the demonstration (OMB control number
0938-1131, now discontinued) and, in March 2012, selected 11 states and the District of
Columbia to participate. Within these states, the participation of 27 private IMDs was approved.
The goal of the 3-year demonstration is to assess whether the expansion of Medicaid coverage to
include services provided in private, free-standing inpatient psychiatric facilities improves access
to and quality of medically necessary care and whether this change in reimbursement policy is
cost-effective. Focusing on psychiatric emergencies, the demonstration is also an attempt to
explore a potential remedy to alleviate one of the factors contributing to psychiatric boarding,
one of the consequences associated with the Medicaid IMD exclusion. The current PRA
submission requests approval to collect data in association with the mandated evaluation of the
demonstration.
Section 2707 of the ACA specifies that the evaluation shall include the following:
1. An assessment of access to inpatient mental health services under the Medicaid
program; average lengths of inpatients stays; and emergency room (ER) visits;
2. An assessment of discharge planning by participating hospitals;
3. An assessment of the impact of the demonstration project on the costs of the full
range of mental health services (including inpatient, emergency, and ambulatory
care);
4. An analysis of the percentage of consumers with Medicaid coverage who are
admitted to inpatient facilities as a result of the demonstration project as compared to
those admitted to these same facilities through other means; and
5. A recommendation regarding whether the demonstration project should be continued
after December 31, 2013, and expanded on a national basis.
The ACA further mandates that “not later than December 31, 2013, the Secretary shall
submit to Congress and make available to the public a report on the findings of the evaluation.”

2

In addition to the requirements for the evaluation, the ACA specifies the following aspects
of the demonstration:
1. The population served by the demonstration is limited to individuals who require
medical assistance to stabilize a psychiatric EMC, defined as a situation where the
individual “expresses suicidal or homicidal thoughts or gestures, if determined
dangerous to self or others.” CMS expanded the eligibility criteria, as of October 1,
2012, to also include Medicaid enrollees who may not have suicidal or homicidal
thoughts, gestures, or ideations but are, nevertheless, determined to be dangerous to
self or others.
2. States are required to establish a mechanism for determining whether or not
participants have been stabilized, meaning that “the EMC no longer exists…and the
individual is no longer dangerous to self or others.” The stabilization assessment
mechanism must commence before the third day of the inpatient stay.
Understanding the manner in which these requirements have been operationalized and the
way in which they may affect outcomes of the demonstration is important for informing possible
continuance and expansion of the demonstration on a national basis.
In addition to the ACA specifications regarding the demonstration, many stakeholder groups
believe that the IMD exclusion, coupled with a general shortage of specialized inpatient
psychiatric beds, contributes to extended psychiatric boarding—the practice of holding a patient
with a psychiatric EMC in an ER or general hospital nonpsychiatric medical unit (known as
“scatter beds”) because no specialized beds are available. Psychiatric boarding is thought to
contribute to overcrowding of ERs and to result in substandard care for beneficiaries in facilities
that are not well equipped to treat psychiatric conditions. The expectation is that by increasing
access to IMDs, the demonstration will decrease psychiatric boarding and use of scatter beds,
thereby improving quality of care for beneficiaries with psychiatric EMCs. Therefore, an
assessment of changes in psychiatric boarding, use of scatter beds, and quality of care is essential
for understanding the extent to which the demonstration results meet stakeholder expectations.
To respond to the ACA evaluation requirements, CMS is planning a comprehensive, mixedmethods evaluation of the MEPD. CMS is requesting approval from the Office of Management
and Budget (OMB) for the collection of qualitative data through site visits and key informant and
beneficiary interviews as part of the evaluation.
Fully assessing all of the areas mandated by the ACA as well as the interests of critical
stakeholders necessitates a mixed-methods approach. Quantitative data on service utilization and
expenditures are critical to successfully evaluating the MEPD’s impact in ACA-mandated
evaluation areas A, C, and D. Only a qualitative approach, however, can provide a full
assessment of discharge planning by participating hospitals, as mandated by ACA evaluation
area B, and of psychiatric EMC determination and stabilization processes utilized to ensure
compliance with ACA demonstration requirements; in addition, few if any data are available on
the use of scatter beds and psychiatric boarding times in ERs, so an understanding of the extent
and impact of these practices may only be possible through qualitative methods.

3

Quantitative Data Collection
To the extent possible, the evaluation will use publicly available data to minimize burden on
the demonstration states and facilities. Medicaid and Medicare enrollment and claims data
submitted to CMS will be used to address ACA evaluation areas A and C. Variation in the
quality, timeliness, and completeness of Medicaid data across states will necessitate that the
demonstration states assist the evaluation contractor to develop a clear understanding of claims
data submitted by their particular states. In addition, because not all data needed to address the
ACA evaluation mandates are included in claims data, the evaluation will ask the states and
facilities to submit relevant administrative data that they collect for other purposes. In particular,
because of the IMD exclusion, data on IMD inpatient psychiatric admissions are not available
through claims data. Information about admissions as a result of the demonstration will be
available through claims that the states submit to CMS for demonstration payment and
monitoring purposes, but data for comparison group admissions will have to be obtained from
state or facility administrative sources. Comparison data for admissions before the demonstration
began are needed to determine the extent to which IMD admissions, lengths of stay, and costs
during the demonstration represent a change from IMD admissions, lengths of stay, and costs
prior to the demonstration. In addition, data from non-participating IMDs are needed to
determine the extent to which such changes are due to the demonstration itself rather than nondemonstration factors. In addition to IMD admissions, identification of psychiatric EMCs may
also not be fully possible through Medicaid and Medicare data alone, and quantitative data on
psychiatric boarding times, if available, must also be obtained directly from states or facilities.
Qualitative Data Collection
Information on processes of care that are critical to the success of the demonstration are not
available through quantitative data. Nonetheless, CMS has an interest in ensuring the proper
conduct of discharge planning to (1) achieve positive health outcomes for Medicaid
beneficiaries, (2) limit costs related to readmissions that may occur when discharges are
premature, and (3) ensure that clients are served in community-based settings whenever possible.
While these outcomes of discharge planning will be assessed through analysis of quantitative
data, information about the processes used to conduct discharge planning itself can only be
obtained through qualitative approaches. Qualitative data are critical for understanding the
relationships among length of stay, initiation of stabilization assessments and discharge planning,
stabilization of emergency psychiatric conditions, and discharge. The qualitative data collected
will enrich the evaluation’s understanding of quantitative results, permit consideration of
alternative explanations for significant changes over time, examine the circumstances under
which varying effects might be expected if Congress expands the demonstration, and help
generate hypotheses about outcomes for further exploration through quantitative data analysis.
Because the demonstration operates at state, facility, and beneficiary levels, CMS proposes a
systematic qualitative data collection approach that addresses each of these levels. Key
informant interviews and document review conducted for each level will be used to crossvalidate one another. Two rounds of site visits will be conducted. The first will occur about 24
months after the start of the demonstration (spring 2014) and focus on admission, stabilization,
and discharge-planning procedures before and after the demonstration. The second round of
visits will occur about a year later (spring 2015) to allow the evaluation team to gather detailed
4

information on changes in these procedures, as well as lessons learned and sustainability of
changes made. For each state, during each round an evaluation contractor will visit all
participating IMDs and, for each IMD, one ER that refers patients to that IMD and one general
hospital that admits patients with psychiatric EMCs to general medical units when no psychiatric
bed is available.
During the site visits, the evaluation contractor will interview facility staff regarding
processes of care that are critical to the success of the demonstration, namely procedures for
psychiatric EMC determination, inpatient admissions, stabilization assessment, stabilization, and
discharge planning. Interview questions for staff at each type of facility are included in
Attachment B. In addition, during each site visit, purposive sampling will be used to select 10
medical records to review at each facility. (See Attachment C for sampling procedures.) From
IMD and general hospital records, information regarding stabilization and discharge-planning
procedures and interventions administered will be extracted. From ER records, CMS proposes
extracting information regarding length of time spent in the ER, psychiatric EMC determination,
interventions administered, and inpatient referral procedures. The medical record review tool is
included in Attachment D. Records from both pre- and post-demonstration time periods will be
reviewed to assess how care has changed.
Prior to each site visit, the evaluation contractor will conduct a semi-structured phone
interview with the state demonstration project director, using questions included in Attachment
B. The contractor will also review site documents, such as operation plans, psychiatric EMC
determination procedures, and stabilization assessment and discharge-planning policies. After
each round of site visits, evaluation teams will conduct telephone interviews with five
beneficiaries receiving inpatient services through the demonstration from each participating
IMD, for a total of 135 interviews. These interviews will be essential to understanding
beneficiaries’ experiences with the admission and discharge processes. Moreover, the
beneficiaries’ viewpoints are critical to understanding if and how quality of care improves as a
result of the demonstration. Beneficiary interview questions are included in Attachment E, along
with a draft consent form and the recruitment script for beneficiary interviews.
To manage the voluminous qualitative data collected from interviews and site documents,
the evaluation contractor will systematically code and analyze the data using Atlas.ti, a
qualitative data analysis software package. Data gathered from medical records will be entered
into a data entry program called Viking. These data will then be exported as SAS files for
analysis at the facility-level.
B. Justification
Need and Legal Basis
Section 2707 of the Patient Protection and Affordable Care Act (ACA) of 2011 (P.L. 111148; Attachment A) directs the Secretary of Health and Human Services to conduct and evaluate
a demonstration project to determine the impact of providing payment under Medicaid for
inpatient services provided by private IMDs to individuals with emergency psychiatric
conditions between the ages of 21 and 64.

5

Information Users
The data will be used by CMS to evaluate the MEPD in accordance with the ACA mandates.
This evaluation in turn will be used by Congress to determine whether to continue or expand the
demonstration. If the decision is made to expand the demonstration, the data collected will help
to inform CMS and their stakeholders about possible effects of contextual factors and important
procedural issues to consider in the expansion, as well as the likelihood of various outcomes.
Although the results of this data collection will not be included in the report to be submitted to
Congress by December 31, 2013, we anticipate that Congressional and stakeholder interest will
continue until the evaluation results are published. A comprehensive report of the findings will
be produced in the final year of the project, and interim results will be described in annual
reports and presentations made via webinar during the final two years of the project.
The conceptual framework for the evaluation is presented in visual and narrative form in
Attachment F. Table 1, below, lists the specific research questions to be examined, how each
relates to the ACA mandates, and the data sources that will be used to answer each question. As
shown in the table, CMS Medicaid and Medicare claims data will be the primary data sources for
addressing ACA-mandated evaluation areas A and C, state and facility administrative data will
contribute importantly to addressing all mandated areas, and the qualitative data sources together
will provide the primary information for addressing ACA-mandated area B (discharge planning),
as well as psychiatric boarding, which falls under area A, regarding ER visits. In addition to
discharge planning, the multiple sources of qualitative data will provide important crossvalidating perspectives on processes of care that are critical to understanding the success or
failure of the demonstration, including psychiatric EMC determination, inpatient admission, and
stabilization procedures.
Table 1. Use of Data to Answer Research Questions Addressing ACA-Mandated Evaluation Areas
Research Question

Data Source

ACA-Mandated Evaluation Area

To what extent do private IMDs
increase admissions of Medicaid
beneficiaries with psychiatric
emergencies as a result of the
demonstration?

CMS demonstration payment and
monitoring data
State or facility administrative data

(A) Medicaid inpatient access,
length of stay, and reduced ER visits

Does the demonstration decrease
admissions to nonpsychiatric units of
general hospitals for Medicaid
beneficiaries with psychiatric
emergencies?

CMS Medicaid and Medicare
claims data—pre- and postdemonstration

(A) Medicaid inpatient access,
length of stay, and reduced ER visits

What is the demonstration’s effect on
lengths of stay for Medicaid
beneficiaries with psychiatric
emergencies admitted to private IMDs
compared with lengths of stay in these
facilities before the demonstration and
to lengths of stay in other facilities?

CMS demonstration payment and
monitoring data
CMS Medicaid and Medicare
claims data
State or facility administrative data

(A) Medicaid inpatient access,
length of stay, and reduced ER visits

6

(D) The percentage of consumers
who are admitted to participating
IMDs as a result of the
demonstration compared to those
admitted to the same facilities with
other payment arrangements

(C) Impact on system costs of the
full range of mental health services,
including inpatient, emergency, and
ambulatory care

Research Question

Data Source

ACA-Mandated Evaluation Area

What is the demonstration’s effect on
lengths of stay for Medicaid
beneficiaries with psychiatric
emergencies admitted to scatter beds
in general hospitals?

CMS Medicaid and Medicare
claims data

(A) Medicaid inpatient access,
length of stay, and reduced ER visits

Are fewer Medicaid beneficiaries with
psychiatric emergencies seen in ERs
as a result of the demonstration?

CMS Medicaid and Medicare
claims data—pre- and postdemonstration

(A) Medicaid inpatient access,
length of stay, and reduced ER visits

Does the demonstration reduce
psychiatric boarding time in ERs for
Medicaid beneficiaries with psychiatric
emergencies?

ER administrative data
Key informant interviews
Beneficiary interviews
Medical records review

(A) Medicaid inpatient access,
length of stay, and reduced ER visits

Does the demonstration increase the
proportion of individuals discharged
with a continuing care plan from the
participating hospitals?

Quality improvement data
obtained from CMS’s Inpatient
Psychiatric Facilities Quality
Reporting program

(B) Discharge planning

Does the demonstration improve the
quality of discharge plans?
- Does the demonstration increase
the length of time spent developing
a discharge plan for Medicaid
beneficiaries with psychiatric
emergencies in participating IMDs?
- Does the demonstration increase
the proportion of Medicaid
beneficiaries with psychiatric
emergencies in participating IMDs
who are discharged to communitybased residences (compared to
before the demonstration and
compared to nonparticipating IMDs
and nonpsychiatric units of general
hospitals)?
- Does the demonstration increase
the level of detail (e.g., appointment
times, names of providers) included
in the discharge plans for Medicaid
beneficiaries with psychiatric
emergencies in participating IMDs?
- How does the discharge-planning
process in participating IMDs
compare (in terms of the previous
questions) to the processes in
nonpsychiatric units of general
hospitals?

CMS demonstration payment and
monitoring data
Key informant interviews
Document review
Medical record reviews
Beneficiary interviews

(B) Discharge planning

(C) Impact on system costs of the
full range of mental health services,
including inpatient, emergency, and
ambulatory care

7

(C) Impact on system costs of the
full range of mental health services,
including inpatient, emergency, and
ambulatory care

Research Question

Data Source

ACA-Mandated Evaluation Area

Does the demonstration reduce 30day readmissions (all cause and
psychiatric) for patients discharged
from participating IMDs for a
psychiatric emergency (compared to
before the demonstration and
compared to nonparticipating IMDs
and nonpsychiatric units of general
hospitals)?
What effect does the demonstration
have on costs to the IMDs, states,
Medicaid, and Medicare?
- What are the federal Medicaid
costs for care provided by private
IMDs as a result of the
demonstration?
- To what extent do costs incurred by
the states for Medicaid emergency
IMD admissions decrease after the
demonstration’s implementation?
- To what extent do costs incurred by
participating IMDs for Medicaid
emergency IMD admissions
decrease after the demonstration’s
implementation?
- What is the demonstration’s effect
on overall costs to Medicaid and
Medicare for care provided to
beneficiaries with emergency
psychiatric conditions (perhaps
through cost savings in ER
utilization, general hospital scatter
bed and inpatient psychiatric unit
admissions, nursing home
admissions, and so forth)?
- What additional administrative
costs are incurred by states and
participating facilities to fully
implement the demonstration’s
service-delivery models?

CMS Medicaid and Medicare
claims data
State or facility administrative data

(B) Discharge planning

CMS demonstration payment and
monitoring data
CMS Medicaid and Medicare
claims data
State or facility administrative data
Key informant interviews

(C) Impact on system costs of the
full range of mental health services,
including inpatient, emergency, and
ambulatory care

CMS demonstration payment and
monitoring data
State or facility administrative data

(D) The percentage of consumers
who are admitted to participating
IMDs as a result of the
demonstration compared to those
admitted to the same facilities with
other payment arrangements

Within participating IMDs, how does
the percentage of inpatients who are
Medicaid beneficiaries admitted as a
result of a psychiatric emergency
change relative to the percentage of
inpatients admitted through other
means (i.e., with payment sources
other than Medicaid and/or not as a
result of a psychiatric emergency) after
the demonstration’s implementation?
How does the process of assessing
stabilization in participating IMDs
compare to the processes used before
the demonstration and to processes in
nonpsychiatric units of general
hospitals?

Document review
Key informant interviews
Beneficiary interviews
Medical records review

8

(C) Impact on system costs of the
full range of mental health services,
including inpatient, emergency and
ambulatory care

ACA Demonstration Requirement for
“Stabilization Review”

Use of Information Technology
States and facilities will submit quantitative administrative data electronically through
secure file-transfer programs or encrypted CD-ROMs. These means are necessary to ensure the
security of the data in transit.
Digital audio recording of all interviews (with respondents’ permission) will be the primary
electronic method for ensuring the completeness and quality of interview data. Recording also
enhances efficiency and reduces respondent burden by allowing researchers to review and edit
their written or typed notes without calling respondents for clarification or to check quotes.
Obtaining high quality data through semistructured interviews requires flexible exchange
and conversational rapport between interviewer and respondent. Although information
technology can greatly enhance the smooth administration of large-scale surveys with complex
skip patterns, in qualitative interviewing, it is often best to avoid complex skip patterns in the
first place. For this data collection, the contractor will minimize the skip patterns an interviewer
must navigate during interviews by customizing the protocols in advance. The interview
protocols accompanying this package have been customized for five types of respondents: state
demonstration project directors; staff of IMDs, general hospitals, and ERs; and beneficiaries. In
addition, the site visit teams will be led by trained, experienced interviewers. The interviewers
will be thoroughly familiar with protocol content so they can readily move back and forth within
the protocol without disrupting the conversational flow or asking questions the respondent has
already answered.
After information collection, researchers will use Atlas.ti, an electronic software program
that enables systematic coding and retrieval of textual data according to a specified scheme.
Data gathered from medical records will be entered into a data entry program called Viking.
These data will then be exported as SAS files for analysis at the facility-level.
Duplication of Efforts
This information collection does not duplicate any other effort and the information cannot be
obtained from any other source.
Because the data to be collected are highly specific to the demonstration, no other relevant
data collection effort currently exists. Semistructured interviews will be used only to collect
evaluation information that cannot be obtained from other sources. Where possible, to address its
research questions, CMS will use existing administrative data and secondary data sources, such
as states’ demonstration payment data submitted to CMS; Medicaid and Medicare enrollment,
claims, and encounter data; and administrative data housed in state data warehouses.
Small Businesses
Many, if not all, of the facilities participating in the demonstration may be small businesses
or entities. The site visit and interview protocols have been designed with an effort to minimize
burden on these entities. Every effort will be made to schedule site visits and interviews at the
convenience of these respondents. Evaluation staff will ensure that visits to each facility last no
9

more than one day. The information being requested will be held to the minimum required for
the intended use.
Less Frequent Collection
If this information is not collected, CMS will not be able to address the ACA-mandated
evaluation areas or have a complete and objective understanding of the impacts of the
demonstration on quality of care and the lives of beneficiaries.
Qualitative data will be collected through site visits and interviews twice during the
evaluation, in spring 2014 through summer 2014 and spring 2015 through summer 2015.
Implementation of a complex demonstration project like the MEPD at multiple sites necessarily
faced obstacles; some sites were as much as four months late in implementing it. Therefore, the
first round of qualitative data collection will occur after all sites are fully operational but early
enough to gather reliable data about practices prior to the implementation of the demonstration.
Operational refinements at the state level are likely as the demonstration progresses; therefore,
the second round of data collection is scheduled to capture these changes and ensure a complete,
nuanced evaluation. Not collecting the information at all would seriously impede CMS’s ability
to answer the questions mandated by the ACA, particularly those regarding discharge planning.
Special Circumstances
This request fully complies with the regulations. There are no special circumstances
associated with this information collection.
Federal Register/Outside Consultation
Federal Register Notice
The 60-day notice to solicit public comments was published in the Federal Register on July
26, 2013, vol. 78, No. 144, pp. 45205-45208 (Attachment J). Public comments were received
from two individuals. These comments and our responses are summarized in Attachment K.
Consultation Outside the Agency
CMS’s evaluation contractor presented an overview of the evaluation plans, including
timelines and requirements regarding data collection, to the state demonstration project directors
on October 24, 2012. The contractor also established a nine-member technical expert panel
(TEP) composed of representatives of IMDs, including those involved in the demonstration;
consumers; and other individuals who regularly work with emergency rooms, community mental
health data and systems, and state mental health and Medicaid authorities. Attachment G lists the
members of the TEP and their professional affiliations. On January 16, 2013, the contractor held
an initial meeting with the TEP via webinar to obtain their feedback on evaluation plans,
including the medical record review protocols and beneficiary interview questions. TEP
members have also agreed to be available throughout the project for individual consultation on
design, measurement, and analytic challenges.

10

Payments/Gifts to Respondents
The TEP recommended that incentives be offered to beneficiaries to participate in the
beneficiary interviews in order to obtain an unbiased sample; otherwise, the TEP suggested, only
beneficiaries who are particularly unhappy with the process are likely to participate. The
evaluation team will, therefore, provide a $20 incentive in the form of a check for each
beneficiary interview. A $300,000 incentive pool has also been established for distribution
among states and facilities to offset the burden of participating in site visits and assisting the
evaluation contractor in obtaining and understanding administrative databases. Incentive
payments will be offered on a state-by-state, as-needed basis to ensure necessary cooperation. It
is likely, for example, that such incentive payments may enhance cooperation from facilities not
participating in the demonstration whose staff time, facility access, or data are needed; such may
be the case, for example, with ERs and general hospitals to which site visits will be made or from
which boarding-time data are solicited. The incentive payment to be provided to each state and
facility will vary, depending on the specific types of data requested, the exact number of states
and facilities from which data are requested, and the specific amount of burden entailed for each
of the respondents to provide the needed data, given variations in their systems.
Confidentiality
Individuals and organizations will be told the purposes for which the information they
provide is collected and advised that any identifiable information provided by them will not be
used or disclosed for any other purpose. The evaluation contractor will comply with CMS
privacy guidelines pertaining to personally identifiable information. We are in the process of
obtaining approval from Mathematica’s Institutional Review Board. If required by individual
participating state or local governments or facilities, internal review board approval will be
obtained before conducting site visits and/or interviews.
Key informant interviews, including both telephone interviews with state demonstration
project directors and interviews with facility staff during site visits, will discuss the procedures
utilized in implementing the demonstration and results. Responses are not seen as containing
private information, but they will be aggregated to the extent possible so individual answers will
not be identifiable. Individual responses may be inferred from individual state profiles and case
study narratives, however, because of the limited number of respondents interviewed per state
and facility (for example, there is only one project director per state). For each respondent, name,
professional affiliation, and title will be collected, but Social Security numbers, home contact
information, and similar information that can directly identify the respondent will not be
collected.
Participants in beneficiary interviews will be advised that their responses will be kept
confidential. Respondents will be given this assurance during recruitment (see Attachment E,
Beneficiary Interview Consent Form and Recruitment Script) and again immediately before their
interview. Further, they will receive assurance that the information being gathered is for
evaluation purposes only. Name, contact information, and other identifying information will be
requested only as needed to contact the individual for the interview and to deliver the incentive
payment. Comments made during the interview will not be linked to individual beneficiaries.

11

During the informed consent process and prior to the interview, all interview respondents
will be asked if they give permission to have the conversation audio recorded solely for the
purpose of filling in any gaps in the research notes. Only the research team will have access to
the recording. The beneficiary will be informed that they may request to listen to the audiotape.
The audiotape will be destroyed after the contents are transcribed no later than 90 days after the
interview. If the respondent does not wish to have the interview audiotaped, the interviewer will
take notes instead. The transcription and interview notes will be maintained in a secure studyspecific electronic folder that only a minimum number of research staff members may access.
To maintain patient confidentiality in the medical record reviews, the evaluation contractor
will use a unique numbering system to identify patients in the sample. The contractor number
will indicate the state, type of facility (IMD, ER, or general hospital), and a two-digit suffix
unique to the patient. IMD patients discharged 30–60 days prior to the site visit will be identified
by suffixes between 21 and 29, and IMD patients discharged 30–60 days prior to the
demonstration by suffixes between 31 and 39. Patients discharged from an ER 30–60 days prior
to the site visit will be identified by suffixes between 41 and 49; patients discharged from an ER
30–60 days prior to the demonstration will be given suffixes between 51 and 59; general hospital
patients discharged 30–60 days prior to the site visit will have suffixes between 61 and 69; and
general hospital patients’ discharged 30–60 days prior to the demonstration will have suffixes
between71-79. Site visitors will receive several prenumbered sample labels for each patient
sampled. Site visitors will attach a label to the applicable roster next to the patient’s name and
will enter the number in the record review data collection protocol. The facility points of contact
will be asked to keep the labeled rosters for six months after the site visit in case questions arise
regarding the record review after the site visit is completed.
Data from both the medical record review and beneficiary interviews will be kept
confidential. The evaluation contractor, Mathematica Policy Research, has established data
security plans for the handling of all personally identifiable information, including administrative
data obtained from CMS, states, and facilities; interview notes, audiotapes, coded interview data,
and data processing for the interviews; and medical records abstractions. These plans meet the
requirements of U.S. federal government agencies and are continually reviewed for compliance
with new government requirements and data collection needs. Such security is based on (1)
exacting company policy promulgated by the highest corporate officers in consultation with
systems staff and outside consultants, (2) a secure systems infrastructure that is continually
monitored and evaluated with respect to security risks, and (3) secure work practices of an
informed staff who take all necessary precautions when dealing with confidential data. All
employees also sign a general confidentiality pledge, included as Attachment H. During site
visits, evaluation researchers will at all times keep notebooks and laptop computers on their
persons or in secure, locked locations. Confidential data are kept in study-specific folders that
only a minimum number of staff members may access. All typed or electronically coded
qualitative data are periodically backed up and preserved on secure media.
Sensitive Questions
Given the nature of the demonstration and its evaluation, beneficiary interview questions of
a sensitive nature concerning the individual’s psychiatric condition, his or her recent and past
psychiatric emergencies, and details of medical treatment, are unavoidable. This information is
12

at the center of the qualitative data and is necessary to conduct the evaluation. Beneficiaries will
be advised of the nature of these questions in advance of the interview and informed that their
participation is strictly voluntary. Beneficiaries will also have the option of declining to answer
specific questions without opting out of the interview as a whole; incentive payments will not be
affected by choosing not to answer particular questions. In the event that a beneficiary becomes
upset during the interview, the interviewer will pause and let them collect their thoughts. The
interviewer will ask the beneficiary if they are okay and if they would like to continue, or if they
would prefer a callback at another time. If the interviewer determines that the beneficiary is a
danger to him/herself (i.e., the beneficiary expresses a plan to harm him/herself or others) the
interviewer will stop the interview and give the beneficiary the phone number for the crisis
hotline. All confidentiality and security procedures described in the prior section will apply to
the sensitive information collected. Solicitation of sensitive information will be limited to only
that needed for evaluation purposes.
Burden Estimates (Hours & Wages)
Table 2, below, shows the estimated burden hours and costs for the respondents’ time to
participate in this evaluation. All 12 states will be visited twice. Each site visit will consist of a
visit to each participating IMD and, for each IMD, to one ER that refers patients to the IMD and
one general hospital that boards patients with psychiatric emergencies in nonpsychiatric general
medical units when no psychiatric beds are available. On average, site-visit teams will conduct
four 60-minute interviews each day at each facility, with one respondent per interview. Site-visit
teams will also conduct medical record reviews of 10 medical records at each IMD, referring ER,
and general hospital they visit. In addition to the site visits, estimates are provided for the
associated project director and beneficiary telephone interviews, site-visit planning time,
assistance with gathering documents to be reviewed, and submitting and assisting the evaluation
contractor to understand needed state and facility administrative data.
The total burden for this evaluation is estimated to be 2,613 hours, and the total cost burden
is estimated to be $111,706.
Burden hour estimates are based on prior experience of the evaluation contractor with
evaluations of a similar nature. Throughout the information collection process, the contractor
will monitor the length of the interviews, comments received from participants and field
interviewers, and the number of individuals who refuse to be interviewed. If this information
indicates that the burden on participants is so great as to undermine the collection of high quality
data, procedures will be revised accordingly. For example, the number of questions asked during
interviews may be reduced. If procedures require revision, the CMS will seek OMB approval to
implement specific changes.

13

Table 2. Estimated Total Burden Hours and Cost Over Three Year

Data Collection Activity/
Respondent Type

14

Site Visit Planning—Facility
Administrator
Site Visit Interview—Facility
Administrator
Site Visit Interview—
Psychiatrist
Site Visit Interview—
Counselor (e.g., social
worker, psychologist)
Site Visit Interview—
Registered Nurse
Medical records
assistance—Registered
Nurse
Obtaining beneficiary
consent to be called by
evaluation staff —Social
Worker
Telephone Interview—
Beneficiary
Telephone Interview—
Project Director
Assistance gathering site
documents—Administrative
Assistant
Facilitation of administrative
data requests—Project
Director
Ad-hoc email/phone
communication to answer
questions about MSIS
data—Data Analyst
Assistance in extracting,
sending, and answering
questions about state or
facility administrative data on
IMD admissions and ER
boarding—Data Analyst

Number of
Respondents
per State
Number of States and/or
and/or
Facilities
Facility

Frequency
of
1
Response

Number of
Responses

Average
Burden Per
Response
in Hours

Total
Burden
Hours

Average
Hourly
Wage

Total Cost
Burden

81 (27 IMDs, 27 general
hospitals, and 27 ERs)
81 (27 IMDs, 27 general
hospitals, and 27 ERs)
81 (27 IMDs, 27 general
hospitals, and 27 ERs)
81 (27 IMDs, 27 general
hospitals, and 27 ERs)

1

2

162

2

324

$84.88

$27,501

1

2

162

1

162

$84.88

$13,751

1

2

162

1

162

$83.73

$13,564

1

2

162

1

162

$32.78

$5,310

81 (27 IMDs, 27 general
hospitals, and 27 ERs)
81 (27 IMDs, 27 general
hospitals, and 27 ERs)

1

2

162

1

162

$33.23

$5,383

1

2

162

3

486

$33.23

$16,150

27 IMDs

25

2

1350

0.2

270

$20.50

$5,535

27 IMDs

5

2

270

1

270

$7.25

$1,958

12 states

1

2

24

1

24

$55.04

$1,321

39 (12 states plus 27
IMDs)

1

2

78

0.5

39

$15.87

$619

12 states

1

1

12

2

24

$55.04

$1,321

12 states

1

1

12

2

24

$36.54

$877

12 states

1

2

36

14

504

$36.54

$18,416

3

Data Collection Activity/
Respondent Type
Totals

Number of
Respondents
per State
Number of States and/or
and/or
Facilities
Facility
93 (12 states, 27 IMDs,
27 general hospitals,
and 27 ERs)

Frequency
of
1
Response

Number of
Responses
2,754

Average
Burden Per
Response
in Hours

Total
Burden
Hours
2,613

Average
Hourly
Wage

Total Cost
Burden
$111,706

1

The individuals interviewed during the second round of site visits may differ from those interviewed during the first round, but categories of respondents will
remain the same. Estimates for conversations regarding MSIS data, which will occur on an ad-hoc basis, are for the total number of hours needed over the course
of the three-year evaluation.

2

Assistance will be requested on a schedule to be worked out individually with each state and facility from which data are needed. Average burden per response is
an estimate of time needed for this assistance during each of the three years of the evaluation.

15

Average hourly wages were drawn from the May 2011 National Occupational Employment
and Wage Estimates, United States, as reported by the Bureau of Labor Statistics
(http://www.bls.gov/oes/current/oes_nat.htm, accessed February 18, 2013). Psychiatrist, social
worker, registered nurse, and administrative assistant rates are the average wages for these
positions, respectively. Facility administrator rates were estimated based on wages for chief
executive officers; the rate for counselors was based on the average of wages for social workers
and psychologists, for project directors it was based on general and operations managers, and for
data analysts, it was based on computer programmer wages. The majority of Medicaid
beneficiaries to be interviewed are likely to be unemployed; therefore, the beneficiary rate is
based on the federal minimum wage.
Capital Costs
There are no capital costs.
Cost to Federal Government
Table 3 shows the total and annualized cost for this evaluation. The total cost to the federal
government of the entire evaluation contract is $5,468,458 (including a base period and three
option periods); the annualized cost is $1,367,114 per year. These costs will be incurred from
September 2012 through September 2016.
Table 3. Estimated Total and Annualized Cost for Four-Year Evaluation Contract
Cost Component
Evaluation Design
Data Collection and Analysis
Synthesis of Project Findings
Management and Oversight
Total

Total Cost
$231,159
$4,381,091
$178,379
$677,828
$5,468,458

Annualized Cost
$57,790
$1,095,273
$44,595
$169,457
$1,367,115

Changes to Burden
Based on changes made as a result of the pilot test, the burden estimate has increased since
the 60-day Federal Register notice was published, by a total of 567 hours, from a total of 2,046
hours to 2,613 hours. Correspondingly, the estimated cost burden has increased by a total of
$17,656, from a total of $94,050 to $111,706. The increase results from three sources.
The largest increase is due to the pilot test’s demonstration of the need for facility staff to
assist the evaluation contractor in finding information in the medical records. To reflect this
need, we have added 2.5 hours of staff time to assist with the medical record reviews at each of
the 81 facilities that we visit in each of the two rounds of site visits, for a total increase in burden
of 405 hours, at a cost of $13,458.
A further increase in burden is due to the pilot test’s demonstration of the need to obtain
additional informed consents for beneficiary interviews from IMD staff. We originally asked for
only 10 consents in hopes of interviewing 5 beneficiaries from each IMD. Difficulties in
reaching many of the beneficiaries in the pilot test, however, revealed the need to obtain
additional consents in order to complete five interviews. Therefore, we will now ask staff of each
16

of the 27 IMDs to obtain 25 consents for each of the two rounds of site visits. The IMD visited
during the pilot test indicated that obtaining the consents was easy to do and that obtaining 25
consents would not be a burden to them. The additional consents will require a total of 162
additional burden hours, at a cost of $3,321.
The remaining $877 increase in the burden costs is due to correction of a calculation error in
the original submission.
Publication/Tabulation Dates
CMS expects the site visits to begin in the spring of 2014, pending OMB clearance. CMS’s
evaluation contractor will synthesize the interview data for inclusion in annual reports as well as
a final evaluation report. These reports will integrate qualitative data from the site visits with
quantitative data. The reports will be released to the public only after they have been cleared for
release by CMS. The evaluation contractor will also develop interactive webinar presentations
for key stakeholders that present cross-cutting analyses of integrated quantitative and qualitative
data and provide opportunities for discussion. Webinars will only be scheduled and conducted
upon approval from CMS. Table 4 presents the anticipated data collection, analysis, and
reporting schedule.
Table 4. Schedule of Proposed Data Collection, Analysis, and Reporting
Task

Dates

Qualitative Data Collection and Analysis
Round 1 Site Visits*
Review state documents prior to round 1 site visits
Plan round 1 site visits in collaboration with states and IMDs
Conduct state calls prior to round 1 site visits
Conduct round 1 site visits
Conduct beneficiary interviews after round 1 site visits
Analyze qualitative data from round 1 site visits
Round 2 Site Visits
Review state documents prior to round 2 site visits
Plan round 2 site visits in collaboration with states and IMDs
Conduct state calls prior to round 2 site visits
Conduct round 2 site visits
Conduct beneficiary interviews after round 2 site visits
Analyze qualitative data from round 2 site visits
Quantitative Data Collection and Analysis

Jan.—March 2014
Jan.—March 2014
Jan.—Feb. 2014
March–June 2014
April–July 2014
April–July 2014
Jan.—March 2015
Jan.—March 2015
Jan.—Feb. 2015
March–-June 2015
April–July 2015
April–July 2015

Process Medicaid and Medicare data
Obtain and process state data
Analyze demonstration year 1 data
Analyze demonstration year 2 data
Analyze demonstration year 3 data
Evaluation Reports

Mar. 2013–Nov. 2015
Sept. 2013–Nov. 2015
Mar.–May 2014
Mar.–May 2015
Mar.–May 2016

First annual report
Second annual report
Final report
Webinar presentations

Aug. 29, 2014
Aug. 28, 2015
Sept. 2, 2016
Sept. 2014–Aug. 2016

*The first round of site visits will not begin until the data collection has been approved in accordance with the
Paperwork Reduction Act.

17

Data described in this clearance package will be analyzed to address the research questions
described in Section 1. The first round of site visits and interviews will focus on admission,
stabilization, and discharge-planning procedures before and after the demonstration. The second
round of site visits will take place a few months before the end of the demonstration to allow the
evaluation contractor to gather detailed information on lessons learned, changes in quality of
care, and sustainability. Administrative data submitted by the states and facilities will be used to
supplement Medicaid and Medicare data in analyses of inpatient admissions, emergency
services, and costs.
As noted above, notes from all interviews and document reviews will be typed, uploaded to
Atlas.ti, and coded according to a specified scheme. Analysis of the site visit and interview data
will emphasize policies and procedures that are critical to the implementation of the
demonstration, including psychiatric EMC determination, admissions, stabilization assessment,
stabilization, and discharge planning. The analysis will include identification of themes within
and across states. Throughout the process of gathering, reviewing, and analyzing qualitative
data, quotations will be noted that capture a point of view or an experience particularly well. For
each project, findings from the implementation analysis will be used to interpret findings about
outcomes and to help establish a basis for causal inference. In brief, the interview data collected
under this clearance package, when combined with impact analyses using quantitative
administrative data, will fully address the critical aspects of the demonstration, as mandated by
the ACA.
Expiration Date
This collection does not lend itself to the displaying of an expiration date.

18

PART B
SUPPORTING STATEMENT

COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
Respondent Universe and Sampling Methods
The information collected under this request is not based on probability samples and may
not be generalizable beyond the states included in the demonstration. Interview subjects and
medical records to be reviewed are selected purposively and fall into the following categories:
• State demonstration project directors (one telephone call for each of the universe of
12 states). Calls with the project directors will provide an efficient means for
collecting information on each state’s Medicaid program, mental health delivery
system, implementation procedures, and demonstration successes and challenges. As
the individuals most involved in project design and oversight, state project directors
will provide insight into the implementation of demonstration projects and relevant
contextual factors, and may identify lessons and implications as to the broad
application and sustainability of projects.
• Key informants from each facility to which site visits are made (up to four in-person
interviews at each facility during site visits). Site visits will be made to all 27 IMDs
participating in the demonstration, as well as to one ER that refers patients to each
IMD and, for each IMD, one general hospital that admits patients with psychiatric
EMCs to nonpsychiatric general medical units (that is, scatter beds) when beds in the
IMD are not available. ERs and general hospitals will be selected on the basis of a
review of state demonstration operational plans and conversations with state project
directors and IMD staff about recommended facilities. Priority will be given to
facilities that are active participants in the demonstration and that have the largest
expected impact on or from the demonstration. For example, if the majority of
demonstration referrals to an IMD are made from one particular ER, that ER would
be solicited for the site visit; likewise, general hospitals with particularly high use of
scatter beds prior to the demonstration would be prioritized because the
demonstration aims to alleviate the need for scatter bed use. Because of the need to
understand how the demonstration affects the use of scatter beds, general hospitals
selected may or may not operate acute inpatient psychiatric units. The demonstration
does not alter Medicaid reimbursement for care provided in general hospital
psychiatric units, nor does it aim to divert patients from them or change the care they
provide; therefore, we do not expect to see significant changes in these units as a
result of the demonstration. Funds will be available to provide incentives on an asneeded basis to encourage selected facilities to participate in the site visits. Interview
respondents at each selected facility will include administrators and direct care staff
from each site who are involved in facility operations or who provide direct care to
demonstration participants and can provide information on factors associated with
implementation and outcomes. Administrators may include the chief executive
officer, chief nursing officer, or other senior managers. Direct care providers may
include psychiatrists, registered nurses, and counselors such as social workers and
psychologists. Administrators and direct care providers are important interviewees
because they will provide insight into changes in access to and quality of care due to
the demonstration.
2

• Beneficiaries (five receiving inpatient services from each IMD through the
demonstration, to be interviewed by telephone following their discharge from the
hospital). Beneficiary interviews will be essential to understanding patients’
experiences with the admission and discharge process—for example, the amount of
time spent waiting for admission, their level of involvement in discharge planning,
and how waiting time and participation in discharge planning compare with previous
hospitalizations for psychiatric emergencies. Beneficiary viewpoints are critical to
understanding if and how quality of care improves as a result of the demonstration.
IMD staff will be asked to solicit demonstration participants as they are being
discharged, to procure their consent to be contacted by the evaluation team. All
demonstration participants who are discharged 21 days before the date of the site visit
will be asked to participate until 25 have agreed. Patients will be asked whether they
would be willing to speak with a member of the evaluation team about their
admission and discharge experiences; if they agree to speak with the evaluation team,
the IMD staff member will document contact information for each patient and obtain
signed consent. The consent form will include a discussion of the use of an audio
recording during the interview. The IMD staff will inform patients that they will be
selected randomly for an interview; that is, signing the consent form does not
guarantee that he or she will be called for an interview. Due to logistical
complexities, patients discharged to forensic facilities will not be interviewed. For
patients assigned legal guardians for decision-making purposes, IMD staff will solicit
consent and contact information from both the guardian and the patient. IMD staff
will inform patients that, if selected, they will receive a $20 check from the evaluation
staff for participating in the interview. Across all states, the demonstration is
expected to enroll hundreds to thousands of participants. The 270 beneficiaries
selected for interviews over two rounds of site visits will be selected on the basis of
proximity of their discharge dates to the timing of the site visit. Provision of
incentives will help to encourage participants with a range of experiences to
participate, thereby helping to reduce the potential for bias if only patients with
negative experiences were to respond. Patients with more positive relationships with
the IMD staff soliciting their participation may be more likely to agree to participate.
Due to logistics regarding locating and connecting with individuals for interviews,
patients with more positive discharge experiences (such as those discharged to stable
homes in the community rather than to forensic units, homeless shelters, or other
types of institutional care) may be more likely to participate. Despite these potential
sources of bias, the beneficiary interviews provide an important cross-validation of
information about implementation procedures provided by medical record reviews
and participating facility and demonstration staff, each of which is subject to its own
unique biases.
• Medical records review (10 medical records reviewed at each of 27 IMDs, 27 EDs,
and 27 general hospitals during site visits). Medical records review will cross-validate
and provide a more detailed understanding of stabilization assessment and dischargeplanning procedures, interventions administered to achieve stabilization, length of
time spent in the ER, procedures for determining and documenting the existence of
qualifying psychiatric EMCs, and inpatient referral procedures. Medical records are
important for determining whether the demonstration was implemented as intended,
3

and will facilitate identification of operational lessons learned. Sampling procedures
for medical records to be reviewed are described in detail in Attachment C.
Procedures for the Collection of Information
CMS’s evaluation contractor will use a systematic qualitative data collection approach that
will draw from multiple sources including telephone interviews, document review, beneficiary
telephone interviews, and site visits, including in-person interviews and medical records review.
CMS’s evaluation contractor will conduct two rounds of site visits during the evaluation
period. Pending clearance, the first round will take place about 24 months after the start of the
demonstration (spring 2014 through summer 2014) and will focus on admission, stabilization,
and discharge-planning procedures before and after the demonstration. The timing of the visits
will ensure that states have sufficient time to respond to unforeseen implementation challenges,
and that project procedures operate consistently. The second round will take place a few months
before the end of the demonstration (spring 2015 through summer 2015) to allow the evaluation
contractor to gather detailed information on lessons learned, changes in access and quality of
care, and sustainability. The length of each visit will vary based on the number of IMDs involved
in each state’s demonstration project. Table 5 details the proposed site visit structure and plans
for data collection at each facility.
Table 5. Site Visit Structure and Data Collection
State/Number of Participating IMDs

Days
On Site

Alabama (4)
California (4)

6

Maryland (3)
Missouri (3)
Washington (3)
Illinois (2)
Maine (2)
West Virginia (2)

5

Connecticut (1)
District of Columbia (1)
North Carolina (1)
Rhode Island (1)

3

Note:

3

Site Visit Structure and Data Collection
Structure: The evaluation team will spend one day at each
participating IMD, one day at the IMD’s primary ER referral
source, and one day at a general hospital that admits patients
experiencing a psychiatric emergency to a nonpsychiatric unit
when beds are not available in a psychiatric unit or IMD.
Because of the need to understand how the demonstration
effects the use of scatter beds, the general hospital selected
may or may not operate an acute inpatient psychiatric unit.
Key Informant Interviews: On average, teams will conduct four
60-minute interviews at each facility.
Medical Record Reviews: Teams will review 10 medical records
at each facility: IMD, the IMD’s primary ER referral source, and
the general hospital that admits patients experiencing a
psychiatric emergency when beds are not available in a
psychiatric unit or IMD.

A four-person team will conduct site visits that involve more than one participating IMD. A two-person
team will conduct site visits to states with only one participating IMD.

• State project directors will be interviewed by telephone prior to each round of site
visits. One-hour interviews will focus on identifying any changes in the state’s role
in administering the demonstration and the associated costs, evolving contextual
factors affecting psychiatric emergency and inpatient care in the state, and
implementation facilitators and challenges. The evaluation team will also review
with each project director the state-specific logic model they developed based on
information gathered from document review during the evaluation design phase. The
4

evaluation team will use a standardized set of questions to guide conversations
(Attachment B).
• Direct care staff and administrators from IMDs, ERs, and general hospitals will be
interviewed in person once during each round of site visits. Each interview will last
60 minutes. Semistructured interview guides will indicate the type of information to
be collected but will allow for flexibility across sites in terms of respondents, topics,
and questions asked; this flexibility is critically important given the significant
variation in demonstration projects across states. Attachment B includes a list of the
interview questions to be asked of direct care providers and administrators from
IMDs, ERs, and general hospitals.
• Beneficiary telephone interviews will be conducted with five demonstration
participants discharged from each IMD after each round of site visits, for a total of
135 interviews. Attachment E details the interview questions for beneficiaries and
includes the consent form for beneficiaries and the script IMD staff will use to invite
beneficiaries to participate in the interviews.
• Ten medical records will be selected at each facility (IMD, ER, and general hospital),
using purposive sampling. Attachment C details the sampling procedures. This
technique will enable site visitors to identify records for patients with a wide range of
characteristics of interest, such as high-risk behaviors requiring chemical or physical
restraint, medical comorbidities, or frequent admissions. Direct care staff at each
facility will be asked to assist site visitors in finding information needed for the
evaluation within the medical records; this should take approximately 3 hours for
staff at each facility. Using a structured tool (Attachment D), the evaluation
contractor will abstract from:
-

IMD records, information on stabilization assessment procedures, dischargeplanning procedures, and interventions administered

-

ER records, information on length of time in the ER, EMTALA status
determination, interventions administered, and inpatient referral procedures

-

General hospital records, information on interventions administered,
stabilization assessment procedures, and discharge-planning procedures.

To ensure effective coordination with respondents, the evaluation contractor will use a
systematic approach to communicating and coordinating with IMDs, ERs, and general hospitals.
Table 6 details the sequence of events. Approximately three months before the scheduled site
visit, the contractor will send an email to the demonstration project director and point of contact
for each IMD. The email will describe site-visit activities, identify the approximate time frame
for the visit, and request a date for a planning meeting via telephone to discuss the logistics of
the site visit and all pre-visit activities. During the planning meeting with the IMD point of
contact, the evaluation contractor will discuss the schedule for the site visit to the IMD (for
example, length of interviews with four key informants and time needed for an overview of
medical records) and identify a point of contact for a referring ER and a general hospital that
boards patients with psychiatric emergencies in nonpsychiatric general medical units when no
beds are available in IMDs. The contractor will inform the IMD contact that, on the first day of
5

the site visit, the review team will request two lists of patients from which medical records will
be selected for review.
After the planning meetings with the IMDs are completed, the evaluation contractor will
contact the points of contact at the ERs and general hospitals to discuss site-visit activities and to
schedule interviews with four staff members and time for medical record reviews. For example,
the contractor will ask the ER contacts to provide the team with lists of patients from which
medical records will be selected for review.
Table 6. Site-Visit Planning Protocol
Weeks
Before Site
Visit

Scheduling Activity

Purpose of Activity

12

Send email to demonstration project
directors and IMD points of contact
(POC)

•
•

11

Send follow-up email to
demonstration project directors and
IMD POCs

•
•

Confirm site-visit dates
Confirm IMD POC planning meeting call dates and
times

9–10

Call IMD POCs to plan site visit

•
•

Review site-visit logistics
Discuss site-visit activities and schedule, including staff
interviews and medical record reviews
Request two patient rosters for medical record reviews
Request assistance from IMD staff for beneficiary
recruitment for interviews
Request IMD documents
Identify and obtain contact information for ER and
general hospital POCs

Provide overview of site-visit activities
Propose site-visit dates
• Propose planning meeting telephone call date(s) with
IMD POC(s) during week 10

•
•
•
•
9

Send email to ER and general
hospital POCs

•
•

Provide overview of site-visit activities
Propose site-visit planning meeting telephone call
date(s) and time(s)

9

Send follow-up email to ER and
general hospital POCs

•

Confirm telephone meeting dates and times

7–8

Call ER and general hospital POCs to
plan site visit

•
•
•
•

Review site visit logistics
Discuss site visit activities and schedule, including staff
interviews and medical record reviews
Request two patient rosters from ER POC
Request one patient roster from general hospital POC

•
•
•
•

Confirm any information that might have changed
Provide site-visit team’s names and contact information
Remind POCs about rosters needed from their location
Review site-visit logistics one final time

1–2

Follow up by telephone with IMD, ER,
and general hospital POCs

Quality Control Procedures. Customized, comprehensive training is vital for uniform,
consistently high quality qualitative data collection. The evaluation contractor will conduct two
training sessions in association with each round of telephone interviews and site visits.

6

The training sessions will review the semistructured interview guides, the medical record
review tool, the beneficiary interview guide, and the data coding scheme. The site-visit teams
will practice using the medical record review tool, role-play interviews, and discuss how to
respond to unexpected events while on site. The training will promote reliability in use of the
protocols and will ensure that each contractor staff member shares a common understanding of
the goals of the site visits.
After the first site visit, contractor staff will meet to discuss any changes required to the
interview guides or medical record review tool, with revisions made as needed. Further,
contractor staff will meet after the site visit to review findings and to identify any information
that requires further calls with the site. Once all site visits are complete, the evaluation
contractor will train teams to code qualitative data using Atlas.ti software. The contractor will
follow a thematic coding scheme to be developed by the qualitative research experts (Attachment
I).
The site visit team’s lead will ensure quality and consistency of data collection during the
site visits by conducting reliability assessments to ensure consistent implementation of the
review procedures and accuracy of data collection across team members. At the end of the site
visit, the team’s lead will review all data collection protocols for missing or inconsistent data.
Methods to Maximize Response Rates and Deal with Nonresponse
The interview and medical record data collection is not based on probability samples and is
not meant to represent anyone other than the respondents. Therefore, a response rate does not
apply to these activities. However, in selecting states to participate in the demonstration, CMS
stipulated that states cooperate fully in the cross-state demonstration evaluation. Given this, and
the evaluation contractor’s experience conducting other process evaluations, CMS expects a high
level of participation from state demonstration personnel and facility administrators and direct
care providers. A $300,000 incentive pool has also been established for distribution among states
and facilities to assist the evaluation contractor in obtaining and understanding administrative
databases and conducting site visits. Incentive payments will be offered on a state-by-state, asneeded basis to ensure necessary cooperation. It is likely for example, that cooperation from
facilities not participating in the demonstration from whom staff time, facility access, or data are
needed may be enhanced through such incentive payments; such may be the case, for example,
for ERs and general hospitals to which site visits will be made or from whom boarding time data
are solicited. The incentive payment to be provided to each state and facility will vary,
depending on the specific types of data requested, the exact number of states and facilities from
which data are requested, and the specific amount of burden entailed for each of the respondents
to provide the needed data, given variations in their systems. To further ensure the cooperation
of respondents, contractor staff will attempt to minimize individual burden and develop
interview schedules that respect site constraints and pressures.
•

Minimize individual burden. Willingness of respondents to participate in in-person
interviews may hinge on the time these meetings require. To minimize the burden,
guides are designed to gather information that is as complete as possible in as little
time as possible. The evaluation contractor has developed separate discussion guides
for each respondent type so that respondents are not asked about activities or issues
7

that are not applicable to them. In addition, interviewers will meet with interview
respondents in person in their own offices or at a location of their choice. Telephone
interviews with facility staff will be scheduled at a time that is convenient for the
respondent, and respondents will be provided with the interview questions in advance
to allow them to prepare if they so desire.
•

Develop interview schedules that respect site constraints and pressures. The
contractor will work with each site to determine logistics and a schedule for the inperson interviews. The schedule will avoid conflict with other activities and allow
individuals to find time in their calendars to spend with contractor staff.

Although CMS expects a high degree of participation from all respondent types, direct care
providers may be less readily available for in-person interviews than other respondent types.
The evaluation contractor will offer additional accommodations to this respondent type to
increase the likelihood of their participation. They will offer to meet with direct care providers
outside of clinical hours, restrict the interview to 30 minutes if 60 minutes is not acceptable, and
conduct the interview by telephone if the respondent says that would be more convenient.
To encourage participation of beneficiaries in the interviews, an incentive payment of $20 in
the form of a check will be paid for each interview. This will help to encourage patients with a
range of experiences to participate, thereby helping to reduce potential biases if only patients
with negative experiences were to respond. IMD staff will be asked to solicit agreement from
patients for them to be contacted by the evaluation team as well as their contact information,
immediately prior to discharge. Obtaining contact information at this point will greatly facilitate
the ability to locate discharged demonstration participants; being asked by hospital staff with
whom they are familiar might encourage participation. IMD staff will be asked to obtain consent
to be contacted from 25 patients discharged within 21 days of the site visit; of these 25, only 5
will be randomly selected to be interviewed. Beneficiaries who cannot be located or who chose
not to be interviewed when contacted by the evaluation team will be replaced from among the
remaining pool of those providing initial consent at discharge. The proximity of the interviews
to the respondents’ hospital discharge dates will facilitate the evaluation team’s ability to locate
potential participants and the respondents’ ability to recall details of their recent hospitalization
experiences. Interview respondents may choose not to answer specific questions without
consequences; the interview notes will record such decisions.
Test of Procedures or Methods to Be Undertaken
The evaluation contractor pilot tested the protocols by conducting a site visit and associated
interviews for the Connecticut demonstration project from May 20-22, 2013. Connecticut was
selected because of its proximity to the evaluation contractor’s offices and because only one
IMD is participating in the state’s demonstration project, which simplified logistical
arrangements and allowed the visit and interviews to be completed on an expedited schedule
without violating the PRA.
CMS’s objectives during the pilot test were to assess whether (1) planned procedures allow
collection of the information needed in the allotted time, (2) respondents can readily understand
and answer the interview questions, (3) interviews flow sensibly from topic to topic, and (4) the
questions seem to yield thoughtful, candid responses. The pre-tests were also useful for
8

identifying interviewer training needs and considering refinements to site-visit planning
procedures (for example, how best to identify facilities to visit, individuals to interview, and
medical records to review). Pilot testing also helped confirm the burden estimates.
The site visit protocols attached to this supporting statement (Attachments B - E) directly
reflect the pretest results. Below, we summarize changes made in response to lessons learned
during the pilot test.
•

The medical record review protocol for the IMD took longer than anticipated. To
eliminate redundancy and reduce burden on staff, the contractor removed several
items and subquestions from the medical record review protocols and simplified and
reworded others.

•

Facility staff had lists of medical records prepared for site visits as requested, and
staff noted that preparing these lists required little effort. Procedures for sampling
these records were simplified to reduce burden on facility staff. Rather than asking
staff to recall patient characteristics, the evaluation team will now utilize readily
available information on diagnostic codes and length of stay to select participants. In
addition, the evaluation contractor initially planned to review two open medical
records for patients in the IMD but some information for patients currently receiving
treatment at the IMD in the pilot site was maintained on the unit rather in the medical
records office. To minimize burden on staff working on the unit, the evaluation
contractor will now review only closed medical records during the site visit.

•

The point of contact for the pilot site reported that requesting IMD staff to recruit
beneficiaries to participate in interviews was feasible and that the process was
straightforward and did not require much time. The IMD gave the evaluation
contractor 12 signed consent forms, even though only 10 were requested. Subsequent
to the site visit, however, the contractor had difficulty contacting many of the
beneficiaries who provided consent. To obtain enough consents to ensure that that the
desired sample size can be achieved, therefore, the contractor will now ask IMDs to
gather at least 25 consents.

•

The interview questions were generally understood by respondents, but the majority
of respondents were not aware that the state is participating in the MEPD. Therefore,
questions that refer to changes since the demonstration began were reworded using
the date of implementation as the reference point for changes. Respondents did not
generally have difficulty answering the interview questions, but some clarifying
modifications were made to the interview protocols.

Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data
CMS contracted with Mathematic Policy Research to design and conduct the evaluation of
the MEPD. Table 7 identifies the individuals at Mathematica involved in designing, overseeing,
and analyzing the data.
Mathematica also consulted with a technical expert panel on the methods and data collection
procedures used in this project. Attachment G identifies members of the technical expert panel.
9

Table 7. Individuals Consulted on Statistical Aspects of the Design
Name

Project Role

Email

Phone

Project Management
Crystal Blyler
Quantitative Team

Project Director

[email protected]

(202) 250-3502

Melissa Azur
Jonathan Brown

Team Leader
Senior Researcher/Data Team
Leader
Researcher
Research Analyst

[email protected]
[email protected]

(202) 250-3518
(202) 264-3446
(202) 552-6401
(202) 250-3556

Priyanka Anand
Jessica Nysenbaum
Brenda Natzke
Frank Yoon
Tom Bell
Bryan Bernecker
Lucy Lu
Qualitative Team

Research Analyst
Statistician
Principal Program Analyst
Senior Programmer
Systems Analyst

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Angela Gerolamo
Jung Kim
Rosalind Keith
Grace Ferry
Benjamin Fischer
Jennifer McGovern
Nikkilyn Morrison
Amy Overcash
Other

Team Leader
Researcher
Researcher
Researcher
Program Analyst
Survey Specialist
Survey Specialist
Research Analyst

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

(609) 945-3345
(609) 936-3253
(609) 716-4397
(202) 250-3571
(312) 994-1047
(609) 275-2200
(312) 994-1048
(609) 750-2009

Carol Irvin

Senior Advisor/ Quality
Assurance
Senior Advisor/Quality Assurance
Senior Researcher/Reports

[email protected]

(617) 301-8972

[email protected]
[email protected]

(202) 484-4520
(202) 264-3455

Jim Verdier
Bonnie O’Day

10

(202) 484-3287
(202) 554-7518
(312) 994-1010
(617) 674-8370
(202) 554-7578