SUPPORTING STATEMENT
Part A
Consumer Research on Public Reporting of Hospital Quality Measures
Version 2
January 28, 2011
Division of Consumer Assessment & Plan Performance (DCAPP)
Centers for Medicare & Medicaid Services (CMS)
Table of contents
A. Background 1
B. Justification 4
1. Need and Legal Basis 4
2. Information Users 4
3. Use of Information Technology 5
4. Duplication of Efforts 5
5. Small Businesses 5
6. Less Frequent Collection 5
7. Special Circumstances 5
8. Federal Register/Outside Consultation 5
9. Payments/Gifts to Respondents 6
10. Confidentiality 6
11. Sensitive Questions 7
12. Burden Estimates (Hours & Wages) 7
13. Capital Costs 9
14. Cost to Federal Government 9
15. Changes to Burden 10
16. Publication/Tabulation Dates 10
17. Expiration Date 10
18. Certification Statement………… …………..…………………………………..10
Attachments
One of the primary missions of the Centers for Medicare & Medicaid Services (CMS) is to improve the quality and efficiency of care in the Fee-for-Service (FFS) program. One of the several vehicles used for this mission is the public reporting of quality, efficiency and cost information about hospital care on the Hospital Compare website. This vehicle also serves to provide Medicare beneficiaries and other consumers with the type of data needed to make informed decisions about which providers (in this case, hospitals) to use for their care.
Addition of New Quality Information
Organization and Display of New Quality Measures
CMS often adds important new information about quality of care provided in hospitals on the Hospital Compare website. As new quality measures are added, the website programmers are faced with deciding how to best integrate the new information into the existing website so that it can be easily accessed and understood by consumers. In 2011, CMS will be adding several new measures to Hospital Compare. These new measures will introduce new topics and concepts for consumers and will also use new types of data, than what currently exists on the website. New topics will include: individual measures and composite measures around patient safety, hospital acquired conditions and healthcare associated infections; individual measures around timelines of care provided in the emergency department; and individual and composite measures around the effectiveness of care for heart attack, heart failure, and surgical care. The addition of these new measures raises concern for CMS in how consumers will interpret and be able to use the information. To address these concerns CMS would like to conduct formative research with consumers around the most effective way to incorporate this new information into the website.
Healthcare Associated Infections and Hospital Acquired Conditions
As mentioned above, CMS plans to add new quality information to the Hospital Compare in 2011. Some of that information will include patient safety measures in the area of hospital-acquired conditions (HACs). The introduction of this information comes on the heels of recent legislation around Medicare reimbursement for hospital care. Specifically, the Deficit Reduction Act (DRA), which was signed in February of 2006, contains language creating a system for quality adjustment of Medicare payments for inpatient hospital services. The law required the Secretary of Health and Human Services (HHS) to identify at least two hospital-acquired conditions (HAC) which could have reasonably been avoided through the application of evidence based guidelines and would be subject to the adjustment in payment. On July 31, 2008, in the Inpatient Prospective Payment System (IPPS) Fiscal Year (FY) 2009 Final Rule, CMS included 10 categories of conditions that were selected for HAC payment provision. Similarly, based on HHS’ priority on reducing Healthcare Associated Infections (HAIs), CMS published its final rule in July 2010 for Medicare reimbursement for hospitals, which included the provision that hospitals would start reporting certain HAIs. This reporting will be a part of the CMS Hospital Inpatient Quality Reporting Program, an initiative intended to equip consumers with quality of care information to make more informed decisions about their health care, while encouraging hospitals and clinicians to improve the quality of inpatient care provided to all patients.
Consumer Research
Prior to publicly reporting new information in 2011, CMS proposes to conduct consumer research to ensure that the information is presented in a way that is consumer friendly and understandable for consumers to use when making health care decisions. CMS has contracted to conduct exploratory or formative research around how to best organize and display the new quality information, to assist CMS in developing consumer friendly displays of the information and to cognitively test some of the specific measures to be added to Hospital Compare.
Formative/Exploratory Research
Given that the information presents new concepts around healthcare associated infections (HAIs) and hospital acquired conditions (HACs), information which does not currently exist on Hospital Compare, CMS proposes to conduct a round of formative research to explore consumers’ perceptions, knowledge and understanding of this new topic and to also explore options for effectively integrating the new information into the website, through the existing organizational scheme or potentially through a revised schema. The research team will conduct the exploratory research using focus groups. Focus groups are an effective tool for eliciting individuals’ perspectives and examining the differences in their opinions and reactions. Focus groups will allow the research team to gather baseline information from a cross section of potential users, get multiple points of view at one time, and stimulate discussion regarding perceptions and attitudes around the topics, and explore options for language, display, and navigation through group discussion and brainstorming.
Specifically, the research team will conducted a series of focus group discussions with consumers and caregivers. The consumer participants will range in age from 40 to 70 years of age and will include persons with a mix of hospital experience (persons who have had a recent hospital stay and persons who anticipate a hospital stay in the near future); the caregiver participants will range in age from 18 to 70 years of age and provide care for persons who have had a mix of hospital experience.
Cognitive Interviews
Subsequent to the exploratory or formative research, the research team proposes to design and cognitively test mock-ups and explanatory language around the new measures. As the decision on which measures will be added to Hospital Compare and exactly when has not been finalized, we are using an example of new HAC and HAI measures - Central Line-Associated Bloodstream Infections (CLABSI) – as an example of proposing how the research team would conduct the cognitive interviews. Using this example, a moderator will explore how the research subjects cognitively approach the topic of HAIs/HACs and how they react to the new information.
The goal of cognitively testing the mock-ups will be to determine how to best integrate the new information into the existing website using the display devices similar to those used for extant measures, but presenting the information clearly and in such a way that consumers could draw accurate and useful inferences from the data.
The team proposes using one-on-one cognitive interviews for this research, in order to gain more in-depth feedback from individuals. Cognitive interviews can yield detailed, more nuanced information because of the focused time and rapport established with an individual. Our team often uses cognitive testing to understand how individuals understand, process, and respond to specific materials. We have found that issues related to comprehension or interpretation of complex technical information is better probed in one-on-one settings, as the dynamics of focus groups can make it difficult to tease out variations among individuals. If possible, the team would conduct two rounds of cognitive testing of the new information, utilizing a second round to fine tune the mock-ups and to make useful recommendations to the website programmers.
Specifically, the research team will conducted in-depth interviews with consumers, caregivers and hospital professionals. The consumer participants will range in age from 40 to 70 years of age and will include persons with a mix of hospital experience (persons who have had a recent hospital stay and persons who anticipate a hospital stay in the near future); the caregiver participants will range in age from 18 to 70 years of age and provide care for persons who have had a mix of hospital experience; and the professional participants will include hospital providers with some knowledge and/or experience with these types of infections.
The purpose of the consumer research will be to gain feedback from subjects who may reflect the population of users of the website, without attempting to definitively determine how any particular population of users would react to and use the site. That is, the team is not attempting to achieve an understanding that can be generalized, so much as to identify particular issues or concerns that should be avoided in the presentation of the data. Because this study is qualitative, the data collection will employ convenience samples of individuals from different segments of the population. An estimated breakdown of research participants is displayed in exhibit 1.
Exhibit 1.
Research Round/Participant Type |
Round 1 Formative/Exploratory Focus Groups |
Round 2 Cognitive Interviews |
Round 3 Cognitive Interviews |
Consumers who anticipate having a hospital stay within the next six months |
20 |
6-8 |
6-8 |
Caregivers of consumers who anticipate having a hospital stay within the next six months |
20 |
6-8 |
6-8 |
Hospital professionals with some knowledge and/or experience with these types of infections |
|
6 |
6 |
Total Research Participants |
40 |
20 |
20 |
We will work with established, reliable market research firms to enlist the necessary participants, providing them tailored screeners for the recruitment process. We have developed a recruitment screener that includes a detailed script along with specific criteria to use when contacting potential participants. The script includes screening questions designed to target the specific audience required for this research, including such criteria as gender, age, race and ethnicity, and level of education. Our screening tool also takes into account certain exclusion criteria, screening out, for example, individuals who work in the health care industry, to avoid response bias.
This project consists of the following data collection tools:
1) Recruitment screening forms to be used by the research facilities to match the participant pool to the recruitment goals of the research team (see Attachment A);
2) Semi-structured interviews with patients, caregivers, and hospital professionals (see Attachments B); and
3) Mock ups (see Attachment C).
This project will support CMS’s efforts to understand how diverse populations react to and utilize consumer information. The research shall be conducted in three different sites, to include locations with diverse population characteristics and varying levels of hospital choice. We propose conducting this research from May through June, in order to meet Agency deadlines for adding the new information to Hospital Compare by December 2011.
Justification
As noted above, CMS is committed to being transparent and to making publicly reported information useful to consumers. The work under this collection of information is needed to insure the new rates of hospital acquired conditions and healthcare-associated infections and other information meet those goals.
The information that is gathered during the formative research will be used to develop draft displays of the new quality information and accompanying explanatory information for consumer use. The research team will create paper-based mockups of the information to be tested with research participants, during the second and third rounds of research. The information that is collected from those interviews will be used to revise mockups of displays and explanatory language for a final round of consumer testing. Areas that are of concern from testing will be presented with options and pros and cons for leadership decision-making in a research report. For example, if two different display options have both advantages and disadvantages, the report describing results of this research shall attempt to provide leadership with a basis for making a decision. The format of the report will include screenshots and detailed instructions to help website programmers accurately implement the recommendations.
Each research participant will participate in one focus group or one cognitive interview session. Once the interview data are collected, they will be transcribed. A process of qualitative analysis will be used to identify common themes across multiple audiences. Transcripts, video and audio recordings will be reviewed to insure that no biases were introduced (e.g., from leading questions) and that alternate interpretations of the data are considered.
This data collection will utilize digital recording technology to collect, store, and manage the interview data. Participants will respond verbally to guiding questions.
An extensive review of the literature will be conducted prior to commencing the consumer research. Most of the information is new to public reporting and/or has not been reported as quality information. Hence, while prior research provides some guidance on how to present these data, additional research is needed to insure the data are accurately interpreted and effectively used.
This data collection does not involve or impact small businesses or other small entities.
This is a one-time data collection.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
The 60-day FR notice for this collection published on November 18, 2010. Vol. 75, NO. 223, pg 70930. CMS has committed itself to publicly reporting new hospital quality information on Hospital Compare when it updates the website in December 2011. Workflows for measure developers and website programmers require that consumer testing recommendations be available in June 2011 to meet this deadline.
Therefore, CMS plans to publish the Federal Register notice in November 2010, for a 60-day comment period, prior to submission to OMB. The team will summarize public comments received in response to that notice and describe actions taken by CMS in response to those comments. The team will then complete a revised signature package, based on the public comments received. The team anticipates publishing the revised package in the Federal Register in February 2011, for a 30-day comment period. The team will then revise the research approach based on any additional comments received during the 30-day comment period.
A stakeholder workgroup will be convened by the research team to obtain feedback on the research approach from a diverse set of stakeholders. This workgroup will consist of representatives of consumer advocacy organizations, hospital associations, physicians, foundations that support public reporting of healthcare quality and quality improvement, business associations, insurers and other purchasers.
Cognitive interview subjects will receive a stipend, with the amount varying depending on the difficulty of recruiting subjects of different characteristics. For example, the research team knows from previous experience that hospital professionals are particularly challenging to recruit. Also, the team has identified specific criteria for recruiting consumers and caregivers that brings relevant experience and knowledge for the specific research goals and objectives.
The team has completed research with similar audiences and have learned that in order to ensure an adequate response rate, or in the case of this research to ensure the participants with the right characteristics are willing to take the time to participate in the research, it is necessary to compensate participants with realistic rates. In order to ensure that hospital providers, for example, are willing to take time out of their workday to travel to and from the research site and participate in the interview, it requires a realistic payment. The team has also learned that there are regional variations in market rates for incentives. Taking into account respondent incentives used in previous research, we propose providing consumers, e.g. beneficiaries or caregivers, a range of $75 to $100 for their participation and hospital professionals a range of $200-$300.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Information that can directly identify the respondent, such as name and age, will be collected by the research facilities. However, only the age and first name of each participant shall be passed on to the CMS research contractor. Participant confidentiality will be protected by de-identifying audio data upon transcription. All data will be stored on encrypted and password protected files. Data will only be presented in aggregate and de-identified format. Only the principal investigator and research staff necessary to conduct research will have access to the data. All research data is maintained in locked cabinets within a locked data storage area. All electronic files are password and encryption protected. All activities stated in this project will be performed in concordance with the Health Insurance Portability and Accountability Act (HIPPA) Privacy Rule, 45 CFR Parts 160 and 164. Informed consent will be obtained from all who participate in the semi-structured interviews. (See Attachment D).
Interviews will be conducted in private with an individual focus group moderator or interviewer and will focus on soliciting feedback on the quality information and measure displays, rather than on any personal or sensitive information. However, interviewees appearing distressed or confused by the interview shall be provided any additional information or a referral to medical or counseling help as needed. Interviewees who wish to investigate CMS consumer information further will be given links and/or phone numbers as necessary.
Exhibits 2a, 2b and 2c show the estimated annualized burden hours for the respondent's time to participate in this project. The screening phone call will be experienced by all participants and is expected to take approximately 5 minutes to complete. It is estimated that half of those called by the research facility will be selected for the research study. Focus group discussions will last about 1.5 hours for consumers; cognitive interviews will last about 1.5 hours for consumers and 1 hour for professionals. Exhibits 2a, 2b and 2c show the estimated annualized burden for the respondent’s time to participate in the project The total burden for all participants in the first round of research is estimated to be 127 hours; the total burden for all participants in the second round of research is estimated to be 57 hours; and the total burden for all participants in the third round of research is estimated to be 57 hours.
Exhibits 3a, 3b and 3c show the estimated annualized cost burden for the respondent’s time to participate in the project. The total cost for round one research is estimated to be $2,647; for round two the costs are estimated to be $2532; and for round three the costs are estimated to be $2532.
Exhibit 2a. Estimated annualized burden hours – Round 1 – Exploratory Research
Data Collection Mode |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Screening form |
80 |
1 |
0.083 |
6.64 |
*Focus group discussions with patients and caregivers (4 groups, with 10 participants per group) |
40 |
1 |
3 |
120 |
|
|
|
|
|
Total |
120 |
|
|
126.64 |
* Includes 1.5 hours for research and 1.5 hours assumed time required to travel to and from focus group facility.
Exhibit 2b. Estimated annualized burden hours – Round 2 – Cognitive Interviews
Data Collection Mode |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Screening form |
40 |
1 |
0.083 |
3.33 |
*Semi-structured interviews with patients and caregivers |
14 |
1 |
3 |
42 |
**Semi-structured Interview with hospital professionals |
6 |
1 |
2.5 |
15 |
|
|
|
|
|
Total |
64 |
|
|
57 |
* Includes 1.5 hours for research and 1.5 hours assumed time required to travel to and from focus group facility.
** Includes 1 hour for research and 1.5 hours assumed time required to travel to and from focus group facility.
Exhibit 2c. Estimated annualized burden hours – Round 3 – Cognitive Interviews
Data Collection Mode |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Screening form |
40 |
1 |
0.083 |
3.33 |
*Semi-structured interviews with patients and caregivers |
14 |
1 |
3 |
42 |
**Semi-structured Interview with hospital professionals |
6 |
1 |
2.5 |
15 |
|
|
|
|
|
Total |
64 |
|
|
57 |
Exhibit 3a. Estimated annualized cost burden – Round 1 – Exploratory Research
Data Collection Mode |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Screening form |
80 |
6.64 |
$20.90 |
$139 |
*Focus group discussions with patients and caregivers (4 groups, with 10 participants per group) |
40 |
120 |
$20.90 |
$2,508 |
Total |
120 |
126.64 |
|
$2,647 |
* Hourly wage based on the mean hourly wage estimates from the May 2009 National Occupational Employment and Wage Estimates, U. S. Bureau of Labor Statistics.
Exhibit 3b. Estimated annualized cost burden – Round 2 – Cognitive Interviews
Data Collection Mode |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Screening form |
40 |
3.33 |
$20.90 |
$70 |
Semi-structured interviews with patients and caregivers |
14 |
42 |
$20.90 |
$878 |
*Semi-structured interviews with hospital professionals |
6 |
15 |
$105.66 |
$1,585 |
Total |
60 |
30.33 |
|
$2,532 |
*Provider hourly wage based on mean wage estimates for surgeons from the May 2009 Occupational Employment and Wages, Occupational Employment Statistics, U.S. Department of Labor, Bureau of Labor Statistics.
Exhibit 3c. Estimated annualized cost burden – Round 3 – Cognitive Interviews
Data Collection Mode |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Screening form |
40 |
3.33 |
$20.90 |
$70 |
Semi-structured interviews with patients and caregivers |
14 |
42 |
$20.90 |
$878 |
*Semi-structured interviews with hospital professionals |
6 |
15 |
$105.66 |
$1,585 |
Total |
60 |
30.33 |
|
$2,532 |
There are no direct costs to respondents other than their time to participate in the study.
Exhibit 4 shows the estimated cost to the Federal government for this six month project. The total cost is $204,000. This amount includes all direct and indirect costs of the design, data collection, analysis, and reporting phase of the study.
Exhibit 4. Estimated Cost
Cost Component |
Estimated Total Cost |
Project Development |
15,000 |
Data Collection Activities |
75,000 |
Data Processing and Analysis |
24,000 |
Reporting of Results |
15,000 |
Project Management |
15,000 |
Overhead |
60,000 |
Total |
204,000 |
This is a new collection of information.
Timeline |
2010 |
2011 |
|||||||||||||||
Q4 |
Q1 |
Q2 |
Q3 |
Q4 |
|||||||||||||
Nov |
Dec |
Jan |
Feb |
Mar |
Apr |
May |
Jun |
Jul |
Aug |
Sep |
Oct |
Nov |
Dec |
||||
CMS submits OMB "Clearance Package" to OMB |
X |
|
|
|
|
|
|
|
|
|
|
|
|
|
|||
Federal Register Notice (60 day comment period) |
X |
|
|
|
|
|
|
|
|
|
|
|
|
|
|||
CMS revises OMB “Clearance Package” t o OMB |
|
|
X |
|
|
|
|
|
|
|
|
|
|
|
|||
Federal Register Notice (30 day comment period) |
|
|
|
X |
|
|
|
|
|
|
|
|
|
|
|||
Consumers Research |
|
|
|
|
|
X |
X |
X |
|
|
|
|
|
|
|||
Publication of data on Hospital Compare |
|
|
|
|
|
|
|
|
|
|
|
|
|
X |
|||
CMS would like to display the expiration date. It does not seek an exemption.
Not applicable.
ATTACHMENTS
Attachment A – Screening forms
Attachment B – Cognitive interview guides
Attachment C – Testing Materials/Mockups
Attachment D – Consent form
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |