CMS-10360.REVISED - Supporting Statement Part B

Supporting Statement – Part B
Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling
or other respondent selection method to be used. Data on the number of entities (e.g.,
establishments, state and local government units, households, or persons) in the universe covered
by the collection and in the corresponding sample are to be provided in tabular form for the
universe as a whole and for each of the strata in the proposed sample. Indicate expected response
rates for the collection as a whole. If the collection had been conducted previously, include the
actual response rate achieved during the last collection.

It should be noted at the outset that this data collection is intended to gather only qualitative data.
No survey is being prepared and no experimental design is planned as a follow-up phase. The
intensive interviewing around language and displays of new hospital measures, and especially the
focus group discussions about how to organize measures on Hospital Compare and introduce
consumers to new concepts about quality, use flexible probes and generate open-ended responses.
Furthermore, the population samples used will be relatively small. Hence the data generated are
suitable for qualitative analytic techniques rather than quantitative ones. This supporting statement
will discuss validity and reliability issues as they apply to qualitative research. No statistical
analytic methods per se are anticipated.
The research team will conduct research on 3 target audiences for Hospital Compare, screeners for
which are included in Appendix A. Physicians will be recruited who reflect physicians who are
clinically active and practicing in the community and who make referrals to hospitals. These
physicians are likely to be interested in the performance of hospitals in their community and to be
the experts whom consumers are likely to turn to understand quality reports or to whom they
might express concern about hospital quality when referred for hospital care. The screening tool
script is designed to target clinically active primary care physicians who refer to local hospitals for
care, surgeons, and hospital internists.
The research team will also conduct research on two groups of consumers. Consumers who have
recently had hospital care or who anticipate getting hospital care in the next six months are
potential users of Hospital Compare. In addition, patients may be aided by family members to
make decisions about hospital care and those family members may potentially use Hospital
Compare, making those “informal family caregivers” another consumer audience for the site. We
anticipate recruiting hospital patient consumers, aged 40 to 70 years of age, and informal family
caregiver consumers, aged 18 to70, using established, reliable market research firms and detailed
scripts along with specific criteria to use when contacting potential participants. The script
includes screening questions designed to target the specific audience required for this research,
including such criteria as gender, age, race and ethnicity, and level of education. The script will
screen out those consumers with little or no familiarity with the internet, as Hospital Compare is
an internet-based tool. Our screening tool also takes into account certain exclusion criteria,
1

screening out, for example, individuals who work in the health care or health insurance industries,
to avoid response bias.
We estimate that the telephone screening process will take an average of no more than 5 minutes
per potential recruit for both consumers and physicians. An estimated breakdown of research
participants is displayed in Exhibit 1, each cell representing what we anticipate will be an IC
whose subjects we will draw from the 80 total for this generic submission.
Exhibit 1.

Research Round/Participant Type

Formative/Exploratory
Focus Groups

Intensive
Individual
Interviews

Consumers who have had/anticipate
having a hospital stay within the next
six months
Caregivers of consumers who have
had/anticipate having a hospital stay
within the next six months
Physicians with active clinical
practices.
Total Research Participants for each
set or round of activities
Total Research Participants Overall

20

6-8

Follow-up
Intensive
Individual
Interviews
6-8

20

6-8

6-8

0 [N/A]

6

6

40

20

20

80

2. Describe the procedures for the collection of information including:
-

Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Interviews and focus groups will be conducted at a market research/focus group facility. First the
subjects will learn about the study and be asked to sign their consent for the interview and for
recording of the interview. They will be reassured that none of their responses will be associated
with their name and that their participation will in no way affect their Medicare benefits (if they
are beneficiaries) or relationship with the Medicare program (for physicians who participate in
Medicare). Signed consent forms will be stored separately from any data that are collected from
the participant and will be destroyed no later than within one year after the completion of the
interview or focus group.
Demographic information will be collected by the market research firm and the contractor shall
2

report collective statistics about the sample to CMS. A qualitative review of those statistics will
be conducted jointly by the contractor and CMS staff to determine if plausible alternate
explanations of study results need to be discussed. If the potential participant meets the screening
criteria and would like to participate, he/she is scheduled to participate in a focus group. For those
persons who do not meet the screening criteria or who choose not to participate in the research, no
identifying information will be retained. Should any participant demonstrate unexpectedly
strong emotional distress in response to the materials or the process of the interview or group, they
will be allowed to withdraw and appropriate help or referral will be provided.
Focus groups lasting 2 hours and occurring only once per individual will be used to explore
conceptual frameworks for organizing the various measures currently on Hospital Compare and
soon to be added. Paper diagrams and written descriptions will be discussed in a focus group
setting, facilitated by a professional moderator using a general topic guide. Skilled moderation is
necessary to avoid letting an opinionated member of the group dominate and to solicit honest
responses from less vocal members of the group. Therefore the topic guide is general and leaves
some probing and redirecting to the discretion of the moderator. Probes may be needed to
generate more in-depth discussion and to understand cognitive frameworks underlying the initial
responses to materials.
The specific new measures and how they will be displayed and explained will be explored using
intensive individual interviews, lasting 1-1 ½ hours and occurring only once per individual. To
minimize the impact of fatigue and inattention, interviewers will strive to keep the interviews to 1
hour and will monitor for signs of fatigue and inattention during the interviews. Intensive
interviews combine a period of free exploration with guided attention to certain parts of the
materials; “think aloud” strategies to better understand the thought processes behind certain
behaviors and decisions, as well as very specific and focused questions about interpretation of
specific material. The discussion guide includes standardized questions for all participants, as
well as probes for particular situations, such as when an individual has a very unusual
interpretation of materials. Because the interviews often generate open-ended verbal responses
rather than statistical estimates, results are analyzed using qualitative methodologies.
In addition to the interviewer/moderator, note takers from the contractor staff and CMS staff will
observe interviews and focus groups from behind a two-way mirror in a designated observation
room or through streaming video. Whether in-person or by streaming video, CMS staff
participation insures that strict adherence to the protocol and appropriate probing are conducted by
the contractor. That is, close collaboration between contractor and CMS staff insure quality
control.
CMS will choose the geographic location of the interviews and focus groups with research criteria
in mind (greater hospital choice, local history of recent hospital infections in the news, or other
relevant factors). Follow-up rounds of intensive interviewing will attempt to vary some of the
geographic characteristics so that local idiosyncrasies will not unduly influence the results.
Intensive interviewing samples, however, generally do not aim to achieve full inclusion of all
social and demographic groups. The use of qualitative methods does not preclude basic
tabulations of responses. The tabulations may indicate if the recruit introduced some unintended
3

bias or if alternate explanations of the results found should be considered. However, these
tabulations cannot be used to infer statistical significance or thoroughness of exploration of
challenges communicating to consumers.
3. Describe methods to maximize response rates and to deal with issues of non-response. The
accuracy and reliability of information collected must be shown to be adequate for intended uses.
For collections based on sampling, a special justification must be provided for any collection that
will not yield 'reliable' data that can be generalized to the universe studied.
Previous contractor experience shows that using professional market research/focus group
facilities that maintain large databases of potential respondents is an effective method for
identifying and recruiting appropriate research participants. The contractor will provide the
research facility with specific instructions and closely monitor progress throughout the recruitment
process. Maintaining such close collaboration and oversight enables the contractor and CMS to
identify, and address, potential issues early on. Demographics and other characteristics of the
larger group of potential subjects shall be compared to those of the smaller group of subjects
actually selected for the focus groups and interviews. The contractor and CMS shall study and
report on any inadvertent sample characteristics have been introduced due to the screening criteria
and make any changes to the criteria as needed. If such characteristics are uncovered during or
after a particular round, the research team will make explicit changes to the recruitment and
screening criteria for subsequent rounds.
Both the focus groups and especially the intensive interviews are cognitively demanding and
tiring. Providing participants with monetary remuneration, at levels in accordance with OMB
guidance, will provide an important incentive to participate in the study. Recent efforts by the
contractor to recruit physicians reflective of those who are clinically active in a community for
these types of intensive have proven challenging. To address the challenges of recruiting
physicians reflective of those who are in active clinical practice in the community, the contractor
shall schedule interviews at times convenient to the physician (often before regular work hours).
CMS also may ask for an exception to the standard incentive rate to recruit to recruit such
physicians.
4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an
effective means of refining collections of information to minimize burden and improve utility.
Tests must be approved if they call for answers to identical questions from 10 or more
respondents. A proposed test or set of tests may be submitted for approval separately or in
combination with the main collection of information.
This request for authorization to conduct qualitative research, using a limited sample of research
participants via focus group discussions and intensive interviews, is not for the purpose of
obtaining large amounts of quantitative data but rather for exploring alternate frameworks and
formats for presenting quality information to visitors to the hospital compare website. However,
this qualitative research is similar in nature to an activity called “pre-testing” which is conducted
prior to implementing surveys.

4

The recordings and/or transcripts of interviews and focus groups will be analyzed using qualitative
methods, possibly including qualitative analytic software (such as NUD.IST, NVivo or
ATLAS.ti). Recommended changes (to texts or displays for measures or to the
introductory/organizing framework for quality measures on the website) will be supported by
evidence from the analysis and alternate explanations will be considered. To the extent possible,
modifications will be re-assessed in follow-up rounds with new subjects before final
recommendations are made.
5. Provide the name and telephone number of individuals consulted on statistical aspects of the
design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will
actually collect and/or analyze the information for the agency.
Margaret Gerteis, Ph. D.
Project Lead
Mathematica Policy Research
(617) 301-8990

5