SUPPORTING STATEMENT
Part A
SelectMD 2.0 Clinician Choice Experiment
Version: June 18, 2014
Agency for Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that Make the Collection of Information Necessary 3
2. Purpose and Use of information 5
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 6
6. Consequences if Information Collected Less Frequently 6
7. Special Circumstances 6
8. Federal Register Notice and Outside Consultation 7
9. Payments/Gifts to Respondents 7
10. Assurance of Confidentiality 8
11. Questions of a Sensitive Nature 8
12. Estimates of Annualized Burden Hours and Costs 8
13. Estimates of Annualized Respondent Capital and Maintenance Costs 9
14. Estimates of Annualized Cost to the Government 9
15. Changes in Hour Burden 10
16. Time Schedule, Publication and Analysis Plans 10
17. Exemption for Display of Expiration Date 11
18. List of Attachments 11
The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf) states that the mission of the Agency for Healthcare Research and Quality (AHRQ) is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) program is a multi-year initiative of AHRQ. AHRQ first launched the program in October 1995 in response to concerns about the lack of good information about the quality of health plans from the enrollees' perspective. Numerous public and private organizations collected information on enrollee and patient satisfaction, but the surveys varied from sponsor to sponsor and often changed from year to year. The CAHPS® program was designed to:
Make it possible to compare patient experiences across sponsors and over time; and
Generate tools and resources that sponsors can use to produce understandable and usable comparative information for consumers.
Performance reports on doctors have become increasingly available in recent years, but there is little evidence regarding how consumers understand and use different types of performance information to make choices. The few research studies that do exist on this topic suggest that most consumers pay little attention to standardized quality measures such as CAHPS, clinical process indicators similar to the Healthcare Effectiveness Data and Information Set (HEDIS), or patient safety metrics. There is growing evidence, however, that consumers are interested in anecdotal comments posted by patients online regarding their experiences with their doctors; the number and use of websites with such patient comments has increased rapidly in recent years. Comments from patients often cover the same experiential domains as CAHPS surveys (such as access to care and information, how well clinicians communicate with patients, and the helpfulness and courtesy of office staff), but in ways that can be easier to understand, more engaging, and also more persuasive to those reading a report than statistically summarized survey scores. However, the widespread availability of such anecdotal accounts may distract consumers' attention away from sites that offer less engaging but more reliable measures of quality. Moreover, if sponsors of health care report cards that include CAHPS and other performance measures also incorporate patient comments as a way of attracting users, consumers may have difficulty integrating patient comments with standardized metrics and thus ignore or misunderstand potentially useful information.
This study builds on previous research conducted as part of the CAHPS program to explore new ways of integrating patient comments with other performance metrics in web-based quality reports for consumers to support their choice of physicians. Our previous consumer choice study, referred to as SelectMD 1.0 (approved by OMB on 3/8/10 under OMB Control Number 0935-0161), revealed important risks and opportunities of using patient comments that require additional research in order to develop effective guidance for report sponsors. Sponsors of performance reports in both the public and private sectors, including Federal agencies such as the Centers for Medicare & Medicaid Services (CMS), have indicated strong interest in receiving such guidance on strategies for effectively incorporating patient comments to increase consumers’ use of public reports and to enhance their ability to interpret CAHPS and other performance measures.
This follow-on study (referred to as SelectMD 2.0) will use an experimental design to test different methods of incorporating patient comments along with CAHPS survey results, HEDIS-like measures of effective clinical treatments, and indicators of patient safety in web-based physician quality reports. The study will help AHRQ understand how people choose a doctor as their regular source of medical care and advice. The study has three stages. In the first stage, respondents will be asked some questions about their health care experiences and how they go about choosing a doctor.
In the second stage respondents will log onto an experimental website that has information about a fictitious set of doctors from which to choose. Respondents will be asked to use the information on the website to select a doctor who they think would be the best for their health care needs. Although they will not really be selecting a doctor, they will be asked to consider the choice as carefully as if they were making it for themselves. In the third stage, following their selection of a doctor, respondents will answer a set of questions about how they made their choice of doctor, how useful they found the website, and how confident they were in the choice they made.
This research has the following goals:
to expand on the findings from AHRQ’s previous choice experiment regarding how including narrative patient comments in web-based physician quality reports influences the ways in which consumers learn about and select among clinicians, and
to assess whether and how patient comments can be presented in a way that promotes learning about physician quality and complements rather than detracts from standardized measures of quality.
To achieve the goals of this project the following data collections will be implemented over the three stages of the experiment:
Pre-Choice Survey – The purpose of this survey is to measure the respondents’ previous exposure to information on health care provider performance and how they go about choosing a physician. The pre-choice questionnaire is included as Attachment D-1 and Attachment B presents the study invitation that will be used for recruitment.
Experimental Website – The purpose of this site is to present different combinations and displays of performance information that respondents will use to select a doctor. Respondents will be randomly assigned to one of eight different versions of the experimental SelectMD website that will vary according to the level of detail presented, how patient comments are grouped and labeled, whether respondents can choose which and how much information to review, and whether respondents have access to live telephone assistance when making their choices. A detailed description of the experimental website and the eight versions, including sample wireframes, is presented in Attachment C.
Post-Choice Survey – The purpose of the post-choice survey is to assess how respondents made their doctor selection, how useful the website version assigned to them was in helping to make their choice, and how confident they are in the choice they made. Responses to the post-choice survey will provide insights into which of the experimental website versions are more effective in supporting consumer choice of doctors and why. The post-choice questionnaire is included as Attachment D-2.
This study is being conducted by AHRQ through its contractors, RAND and Yale University, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
The results of this study will be used to develop recommendations for helping consumers to better understand and more effectively use complex information to select health care providers, with the aim of making performance information less burdensome and more accessible, useful, and transparent to the public. In particular, the study findings will inform the design and content of the growing number of web-based reports on provider performance incorporating patient comments along with other measures of quality. By adding to the evidence base on the types and combination of information that are most salient and useful to consumers in choosing among provider options, the study will make a significant contribution to improving current reporting initiatives. In addition, the simulated web-based reports will be made available as examples for other report developers to use. This study is being conducted pursuant to AHRQ’s statutory mandate to promote health care quality improvement by conducting and supporting research that develops and presents scientific evidence regarding all aspects of health care, 42 U.S.C. 299(b)(1), and to conduct research on health care and on systems for the delivery of such health care, 42 U.S.C. 299a.
Participants will complete the experiment through a secure online connection from their homes. Survey data are collected by a web-based survey system (internally referred to as “Dimensions”). Participants take online surveys by using a web-browser to access a unique, secured web URL that is both emailed to them and made available through a secured web-portal. Throughout the interview process, questionnaire data are copied to a secured, centralized database for data processing
Work carried out under this clearance will be designed to reflect specific customer population needs for which the work is being conducted and will not duplicate any other work being done by AHRQ or other Federal agencies.
Respondents are consumers of health care services offered by clinicians and other health care practitioners. The study was designed to minimize burden on all respondents and will not have a significant impact on small businesses or other small entities.
This is a one-time data collection.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on January, 29, 2014 for 60 days (see Attachment G) 01/29/2013 Vol.79 No. 19 pages 4718-4721-4722.
8.b. Outside Consultations
In addition to researchers from RAND and Yale, this study will involve contracts with two outside consulting organizations: 1) Wowza, a small business specializing in website development that will create the experimental website versions described in Attachment C, and 2) GfK, a research company that maintains a representative panel of online U.S. consumers that will manage the recruitment of respondents and the administration of the pre-choice and post-choice surveys.
As noted in 8.b., above, respondents for this study will be recruited by GfK. GfK administers an incentive program in the form of a point system for its panel of online consumers. Points are awarded by GfK to respondents according to the amount of time they spend in completing a study.
GfK has assumed the following incentives in their data collection cost projections for this experiment: 1) points valued at $10 per respondent for 7 of the 8 experimental arms, to ensure that there is a high level of follow-through after the one-week gap between the pre-choice survey and the experiment; 2) points valued at $15 per respondent for the Navigator arm of the experiment, since the time required for this arm may run somewhat longer for each respondent and requires that they schedule a fixed time (appointment with the navigator) rather than complete the second part of the study (SelectMD plus post-choice survey) whenever is convenient for them. These assumptions work out to the following estimate of total incentive payments: 1,400 respondents @ $10 per respondent equals $14,000, plus 175 respondents @ $15 per respondent equals $2,625, for a total sum of incentive payments of $16,625. The incentive payments for respondents participating in the experiment are included in the $79,000 data collection subcontract to GfK. RAND and Yale researchers will offer no direct payments or gifts to respondents for participating in the experiment.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Individuals and organizations contacted will be further assured of the confidentiality of their replies under 42 U.S.C. 1306, and 20 CFR 401 and 4225 U.S.C.552a (Privacy Act of 1974), and OMB Circular No.A-130. In instances where respondent identity is needed, the information collection will fully comply with all respects of the Privacy Act.
There are no questions of a sensitive nature on this survey.
Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this experiment. The portion of the experiment involving respondent participation will take place over a period of approximately two months, once OMB approval has been received. All participants will complete the pre-choice survey, which is estimated to take 10 minutes. To assess the impact of their exposure to the SelectMD website, several questions on the initial pre-choice survey are replicated on the post-choice questionnaire. To reduce the likelihood that respondents will simply repeat the answers that they provided on the pre-choice survey (in an effort to appear consistent), it is essential to allow some time to elapse between the two surveys. Consequently, participants will not have access to the SelectMD website until one week after completing the pre-choice survey. Since we expect that about 5% of participants taking the pre-choice survey will not return to participate in the experiment one week later, the number of respondents initially required is 5% higher (1,575) than the full sample of 1,500 required for the experiment. We estimate based on our previous experience with the SelectMD 1.0 experiment that participants will require about 10 minutes to review the information on the website and select their preferred physician from the set of doctors available. The average time required to complete the post-choice survey is estimated to be 20 minutes. Consequently, respondents will average about 40 minutes completing all three phases of the study.
Exhibit 2 shows the respondents' cost burden for their time to participate in this experiment. The total cost burden is estimated to be $22,297.
Exhibit 1: Estimated Annualized Burden Hours
Form Name |
Number of Respondents |
Number of Responses per Respondent |
Hour per Response (min/60) |
Total Burden-Hours |
|
||||
Pre-Choice Survey |
1575 |
1 |
10/60 |
263 |
Time on Website (Choosing MD) |
1500 |
1 |
10/60 |
250 |
Post-Choice Survey |
1500 |
1 |
20/60 |
500 |
|
||||
Total Hours |
4,575 |
na |
na |
1,013 |
Exhibit 2: Estimated Annualized Cost Burden
Form Name |
Number of Respondents |
Total Burden Hours |
Average Hourly Wage Rate* |
Total Cost Burden |
|
||||
Pre-Choice Survey |
1575 |
263 |
$22.01 |
$5,789 |
Time on Website (Choosing MD) |
1500 |
250 |
$22.01 |
$5,503 |
Post-Choice Survey |
1500 |
500 |
$22.01 |
$11,005 |
|
||||
Total Cost |
|
|
|
$22,297 |
*Based upon the national mean hourly wage for all occupations from the "May 2012 Occupational Employment and Wage Estimates", U.S. Department of Labor, Bureau of Labor Statistics.
There are no direct costs to respondents other than their time to participate in the study.
Estimated Annual Costs to the Federal Government
Exhibit 3 shows the estimated annualized cost of this data collection.
As noted in Section 9, $16,625 accounts for the total cost of participation incentives for study respondents and is included in Data Collection Activities.
The total cost over 3 years of this clearance is $736,747.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
144,089 |
48,030 |
Data Collection Activities |
79,000 |
26,333 |
Data Processing and Analysis |
108,438 |
36,146 |
Publication of Results |
73,058 |
24,353 |
Project Management |
65,775 |
21,925 |
Overhead (Indirect Costs) |
266,387 |
88,796 |
Total |
$736,747 |
245,583 |
Exhibit 4: Annual cost to AHRQ for project oversight
Project Officer – GS 15 Step 5 $141,660 |
5% |
$7,083 |
Health Scientist Administrator GS 13 Grade 5 $101,914 |
5% |
$5,095 |
Program Specialist GS 12 Grade 5 $ 85,703 |
5% |
$4,285 |
Total |
|
$ 16,461 |
Annual salaries based on 2014 OPM Pay Schedule for Washington/DC area: http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2014/DCB.pdf
This is a new collection of information.
The results of this study will be used to develop recommendations for helping consumers to better understand and more effectively use complex information to select health care providers, with the aim of making performance information less burdensome and more accessible, useful, and transparent to the public. The simulated web-based reports will be made available as examples for other report developers to use.
The forecasted timeline is as follow:
Recruit
sample – 30 days from the date of OMB Clearance
Obtain
experimental data – 60 days from the recruitment completion
date
Analyze data – 90 days from the experimental data
collection completion date
Publication summarizing the results –
180 days from the analysis completion date
The plan for analysis includes (a) either analysis of variance or multiple regressions with the study condition (variously coded depending on the particular hypothesis being tested) as the key independent variable, and (b) mediational analyses that can be tested either with SEM or a series of regression analyses.
AHRQ does not seek this exemption.
18. List of Attachments
Attachment A – The Healthcare Research and Quality Act of 1999
Attachment B – SelectMD 2.0 Clinician Choice Experiment Invitation
Attachment C – SelectMD 2.0 Clinician Choice Experiment Overview & Sample Wireframes
Attachment D-1 – Pre-Choice Test Questionnaire
Attachment D-2 – Post-Choice Test Questionnaire
Attachment E – Construction of Patient Comments
Attachment F – Outcome and Process Variables
Attachment G – Federal Register Notice (FRN)
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | erwin.brown |
| File Modified | 2014-06-23 |
| File Created | 2014-06-23 |