SUPPORTING STATEMENT
Part A
Comparative Effectiveness Research – Continuing Education
January 6, 2011
Resubmission: July 27, 2011
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 5
3. Use of Improved Information Technology 6
4. Efforts to Identify Duplication 6
5. Involvement of Small Entities 6
6. Consequences if Information Collected Less Frequently 6
7. Special Circumstances 6
8. Consultation outside the Agency 7
9. Payments/Gifts to Respondents 7
10. Assurance of Confidentiality 7
11. Questions of a Sensitive Nature 8
12. Estimates of Annualized Burden Hours and Costs 8
13. Estimates of Annualized Respondent Capital and Maintenance Costs 9
14. Estimates of Annualized Cost to the Government 9
15. Changes in Hour Burden 9
16. Time Schedule, Publication and Analysis Plans 10
17. Exemption for Display of Expiration Date 12
List of Attachments 12
References 12
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Previous dissemination efforts in health care research and evidence through comparative effectiveness funded by the Federal Government have largely been focused in academic settings, which have limited awareness of and utility of comparative effectiveness research among physicians and clinicians in health care delivery settings. This project implements and evaluates methods that extend beyond the academic setting to engage the target audiences in the health care environment where decisions are typically made.
Most clinicians are required to complete continuing medical education (CME) accepted by accrediting organizations recognized by State medical boards. Over sixty boards require anywhere from 12 CME credits to 50 CME credits per year for a clinician to retain their State licensure. (State Medical Licensure Requirements and Statistics, 2010 www.ama-assn.org/ama1/pub/upload/mm/40/table16.pdf.) AHRQ currently provides CME credits on some of its comparative effectiveness research reviews; however, these CME credits are applicable to physicians only and AHRQ is not conducting any follow-up surveys with physicians on these CME activities to ascertain physician self-reported attitudes and use of comparative effectiveness research. AHRQ is expanding its continuing education to include nurses, nurse practitioners, physician assistants, medical assistants, pharmacists, respiratory therapists, and other allied health professionals, as well as physicians. In addition, AHRQ wants to assess the impact continuing education has on clinician self-reported attitudes, the perceived value of comparative effectiveness research education, the clinician’s confidence in applying comparative effectiveness research in practice, and whether the clinician used the available clinician and patient guides to counsel patients on treatment and management alternatives.
Dissemination of clinical and research findings to clinicians varies in approach, methods and by target audience. Highly technical and scientific publications are peer reviewed and serve to validate the methods, calculations, analysis and conclusions of studies and research. Typically, scientific journals have a narrowly defined readership and information regarding clinical application of findings is not part of the criteria for manuscript acceptance and publication. AHRQ utilizes this format when submitting manuscripts regarding comparative effectiveness research (CER) information for publication in the Annals of Internal Medicine. However, it is nearly impossible to discern whether the manuscript was read, its effect on the reader, and the likelihood that the reader will utilize the information.
Accredited education is widely accepted as a method for dissemination of research findings, and is provided in various ways, including online, on site, and through audio and video presentations. To earn credit for participation, clinicians must provide contact information, allowing for the possibility of follow-up data collections regarding knowledge attitudes and performance information about the participant. AHRQ has also provided accredited education as a method to disseminate CER findings, and with this project, has reaffirmed the value of CME in dissemination of CER findings and expanded the commitment to provide accredited education for multiple health care disciplines.
This project is part of a larger three-year effort involving five complementary but separate projects. Four projects will focus on different audiences and methods for dissemination of health care related information to create awareness of comparative effectiveness research, and the fifth will evaluate the impact of those four projects. The fifth project will collect data from all four projects to compare and analyze the effectiveness of the four different methods of dissemination. This submission focuses on the delivery of continuing education as Part 1 of the 5-part project.
The goal of this project is to enhance awareness of comparative effectiveness research among clinicians and measure the value and impact of these efforts as self –reported by clinicians.
To achieve this goal the following activities and data collections will be implemented:
Provide continuing medical education (CME) or continuing education units (CE/CEU) through the appropriate accrediting organizations by providing 45 continuing education modules, on specific comparative effectiveness research reports. Of these modules, 36 will be provided online: 6 will be conducted live in physician offices and 3 will be web conferences first conducted live, then archived as 3 online enduring modules The data collection will consist of quantitative and qualitative metrics about usage of the 36 online continuing education modules by physicians, pharmacists, nurses, nurse practitioners, physician assistants, medical assistants, allied health professionals, and other clinicians. This activity is designed to raise awareness of and utility of comparative effectiveness research by providing free and easy access to clinician guides and consumer guides for clinicians and their patients/families to assist in making informed decisions about heath care. OMB clearance for the data collections related to the 9 live modules will be sought under a separate clearance.
The following monthly utilization rates for the online CME/CE/CEU activities will be collected: the number of CME/CE/CEU certificates issued, monthly participation statistics, and the number of clinician and consumer guides ordered. Because all of the CME/CE/CEU activities are online, the utilization rates are automatically collected by the contractor’s computer when the health care professional registers for the activity, participates in the online education, requests continuing education credit for the activity, and orders clinician and consumer guides. Therefore, this activity does not require OMB clearance.
CME/CE/CEU registration data is provided by the health care professional when he or she logs on and registers for a course. The health care professional will provide their name, e-mail address, address (selecting either their home or business address), telephone number, type of discipline, and their practice setting. This data is collected to ensure that the health care professional receives CME/CE/CEU credit for the courses that he or she takes and will be used to implement the AHRQ Online Continuing Education Participant Evaluation described below.
AHRQ will use the Online Continuing Education Participant Evaluation to evaluate the effectiveness and impact of the CME/CE/CEU modules at 60 days, 6 months, and 1 year after completion of the module (see Attachment C). The purpose of this evaluation is to assess the clinicians’self-reported confidence level in applying comparative effectiveness research and , the value of comparative effectiveness research in making, educating, and guiding treatment decisions with their patients.AHRQ also seeks to ascertain whether or not clincians are using the clinician and consumer summary guides as well as any new actions the clinician implemented as a result of the CME/CE/CEU activity. Each of the module evaluation questionnaires will ask the same questions with the only difference being the title of the continuing education module.
This study is being conducted by AHRQ through its contractor, PRIME Education, Inc., pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to: the quality, effectiveness, efficiency, appropriateness and value of healthcare services; quality measurement and improvement; and clinical practice. 42 U.S.C. 299a(a)(1),(2) and (4).
Data collected will be used to assess the utility and effectiveness of the educational module in increasing awareness, measuring self-reported attitudes, and utility of information provided in comparative effectiveness research. Data will provide useful quantitative and qualitative metrics which AHRQ can use to measure the outcomes of the project.
Data will demonstrate application of the information and knowledge gained within the educational setting; and whether newly acquired knowledge was transferred to practical settings, effectively closing the gaps that initially motivated the educational intervention. To compare dissemination methods, the fifth project in this series of projects will use this data to evaluate and compare this method of dissemination with the methods of dissemination used in the other three dissemination projects.
This project will use an electronic platform called a learning management system (LMS) to collect data. All online program learners will pre-register with the LMS before starting an activity. Learners who have had an inperson office visit will register to complete the activity and earn their certificate. Voluntary registration occurs at the time learners wish to receive their CME/CE/CEU credits for participating in the module. The LMS is certified to track, store and provide CME/CE/CEU certificates of credit to health care professionals in accordance with criteria set by national accreditation bodies, including the Accreditation Council for Continuing Medical Education (ACCME); the American Nurses Credential Center (ANCC); the Accreditation Council of Continuing Pharmacy Education (ACPE); and many more. The LMS system is linked to the AHRQ assigned .gov website where the programs are hosted (http://ce.effectivehealthcare.ahrq.gov; the site is password protected). Sixty days after completing a program, and again at 6 months and 1 year, each learner will receive an automated survey via email to which they will respond via return electronic submission.
Each of the CME/CE/CEU modules are unique and newly created. While the modules will meet accreditation standards of the accreditation associations that the modules are intended for, because they are new, there is no existing information on their effectiveness or impact.
This project does not involve small entities.
The 60-day post program self-assessment surveys are required in order for PRIME Education Inc to maintain its CME/CE/CEU accreditation standards with many accreditation bodies (ACCME for physicians; ACPE for pharmacists; AANP for nurse practitioners; ANCC for nurses), although survey response is voluntary by the learner and does not impact his/her ability to keep the CME/CE/CEU certificate that was earned after participating in the activity. The voluntary 6-month and 1-year participant evaluation will be utilized to assess the clinicians’ self-reported confidence level in applying comparative effectiveness research, their assessment of how valuable the research is to the clinician, and their self-reported use of the information in their practice. Less frequent data collection would prohibit the ability to measure changes in confidence, value, and use of the information, which would also affect the evaluation of the larger, encompassing five-project program.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on February 28th, 2011 for 60 days (see Attachment D). Two comments were received; see Attachment E for the full comment and AHRQ’s response.
This project does not include any payments or gifts to respondents.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be informed of the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Information that can directly identify the respondent, such as name and/or social security number will not be collected as part of the evaluation data. Identifiable data will be collected and stored in the LMS for purposes of providing CME/CE/CEU credits, (person’s name, address, discipline and practice setting, and health care license number); however, these will not be linked to the evaluation data. No personal, private, or sensitive information will be obtained or requested during the evaluation. Data to be collected will not include demographics (e.g. age, gender, and race/ethnicity).
The following statement will be provided on the evaluation questionnaire: “Your responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c).”
Data collection and storage are designed for data integrity, confidentiality, and security. The system includes a detailed disaster recovery plan, as well as a plan for storage, maintenance, and destruction of information. This ensures integrity of all files, documents, and records maintained on the server. The confidentiality of all data will be assured by use of secure file servers for data submission. Electronic files will be maintained only on a password protected secure network. Hard copies of the online survey data will not be produced. PRIME’s IT department manages access to the secure servers via password protected authentication, and protects sensitive data through daily tape backups stored at a secure off-site facility. Only selected personnel have access to the data which include: Ron Minerd, BS, Director of Operations, and Richard Moreo, BS, Lead Website Developer.
This project includes no questions of a sensitive nature.
Exhibit 1 shows the estimated annualized burden hours for the respondents’ time to participate in this research. Since the first AHRQ CER modules were released in March 31, 2011, PRIME has issued 1,608 educational certificates of completion for 4 modules, averaging 402 certificates per course in 60 days. We calculated the total number of certificates to be issued by multiplying the number of certificates (402) issued in this first 60 days x 6 (365 divided by 60) to estimate the number of certificates that would be issued per year. Projected over the course of accreditation, we estimate that an average of 2,412 certificates will be issued for each course.
Based on PRIME’s experience conducting interval-based surveys among all disciplines, we expect survey completions for the 60 day and 6 month intervals to be 35% of 2,412 or 844. A 20% completion rate or 2,412 or 482 at the end of one year for a single module is anticipated.
The CME/CE/CEU registration data will be collected from every clinician who takes a CME/CE/CEU module and receives a certificate which takes about 5 minutes to complete. A participant need only complete a registration once to take all CME/CE/CEU modules available. The estimated annualized burden is based on an expected response rate of 2,412 health care professionals ( ) . The AHRQ Online Continuing Education Participant Evaluation will be completed by these 2,412 health care professionals at 3 different points in time after completion of the CME/CE/CEU education module, for each module the clinician receives a certificate, and requires 3 minutes to complete. The total annual burden is estimated to be 309 hours.
Exhibit 2 shows the estimated annual cost burden to respondents, based on their time to participate in surveys for each CME/CE/CEU module. The annual cost burden is estimated to be $635,272.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response survey |
Total burden hours |
AHRQ Online Continuing Education CME/CE/CEU Registration Data |
2,412 |
1 |
5/60 |
201 |
AHRQ Online Continuing Education Participant Evaluation – 60 days |
844 |
1 |
3/60 |
42 |
AHRQ Online Continuing Education Participant Evaluation – 6 months |
844 |
1 |
3/60 |
42 |
AHRQ Online Continuing Education Participant Evaluation – 1 year |
482 |
1 |
3/60 |
24 |
Total |
4,582 |
na |
na |
309 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
AHRQ Online Continuing Education CME/CE/CEU Registration Data |
2,412 |
201 |
$46.54 |
$9,355 |
AHRQ Online Continuing Education Participant Evaluation – 60 days |
844 |
42 |
$46.54 |
$1,955 |
AHRQ Online Continuing Education Participant Evaluation – 6 months |
844 |
42 |
$46.54 |
$1,955 |
AHRQ Online Continuing Education Participant Evaluation – 1 year |
482 |
24 |
$46.54 |
$1,117 |
Total |
117,000 |
13,650 |
na |
$14,382 |
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
The total cost of evaluating 39 CME/CE/CEU activities is approximately $390,000 (10% of the total cost of developing, disseminating, and evaluating the modules).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This is a new collection of information.
Time Schedule
Following OMB approval, data collection will begin for any modules that have reached the 60 day point, and all modules released thereafter. Data collection will be on a rolling basis and will occur 60 days after each learner completes the CME/CE/CEU module. The 60-day evaluations will continue to be released on a rolling basis until 844 responses have been received, accreditation of the CME/CE/CE module expires or on September 29, 2013 (the end of this contract) whichever comes first. The first 6-month evaluation will be released 120 days after the first 60-day evaluation, and subsequent 6-month evaluations will be released on a rolling basis 6 months after each learner completes the CME/CE/CEU module. The 6-month evaluations will continue to be released on a rolling basis until 844 responses have been received, or the accreditation of the CME/CE/CEU module expires, or September 29, 2013 (the end of this contract) whichever comes first.
The first 12-month evaluation will be released 305 days after the first 60-day survey, and subsequent 12-month evaluations will be released on a rolling basis 12 months after each learner completes the CME/CE/CEU module. The 12-month evaluations will continue to be released on a rolling basis until 482 responses have been received, the accreditation of the CME/CE/CEU module expires, or September 29, 2013 (the end of this contract) whichever comes first.
Work on data analysis will begin inupon OMB approval. Measurable outcomes data will be provided at 180 days and 1 year after release of each CME/CE/CEU module. Reports will be submitted to the evaluation contractor following the end of each quarter. The ending date for the collection of information and completion of a report is September 29, 2013.
Publication Plan
Published Manuscripts
AHRQ staff may prepare manuscripts that show data analyses that will identify self-reported changes in health care professionals’ attitudes and applied knowledge (among other outcomes) about comparative effectiveness as a decision tool in patient treatment and management. These manuscripts will be submitted to peer-reviewed journals for publication.
Conference Presentations.
AHRQ may also take the content developed for the peer-reviewed journal and present these findings as either poster sessions or conference sessions at meetings and conferences.
AHRQ Health Care Innovations Exchange.
As deemed appropriate with input from AHRQ, outcomes will be shared with the AHRQ Health Care Innovations Exchange staff. If materials are considered appropriate for sharing in the public domain, materials will be prepared in a 508 compliant format for posting on the website at www.innovations.ahrq.gov.
Analysis Plan
The evaluation of the data will compare changes from baseline (collected during the time the learner participated in the CME/CE/CEU module) to follow-up at the 60-day mark, to reassessment at the 6-month mark and the 12-month mark.
AHRQ will produce descriptive statistics such as means, frequencies, medians, modes, and standard deviations for number of clinicians who completed CME/CE modules, practitioner by specialty type, practice setting by State, number of clinician and consumer guides ordered, the clinicians self-reported value of comparative effectiveness research, and clinicians rating of confidence in applying the knowledge. Other statistical analyses will include chi-square test of independence, correlations, multiple regressions, and analysis of variance (ANOVA and MANOVA). These statistical tests will be used to compare and contrast the different continuing education activities as well as ascertain differences in clinician attitudes and use of the knowledge. For example, comparisons will be done on clinicians’ ratings their learning experience between specialty practices or type of module (i.e., written, video, interactive). Analyses will also be conducted to ascertain if clinicians who indicated they had intent to implement the findings into practice had actually implemented the findings 6 months and 1 year later. In addition, for clinicians who ordered the summary guides, analyses will be done to find out if the clinicians used the guides and distributed consumer guides to their patients.
Below is a summary of the data that will be collected and analyzed for this project:
Participant Demographics
Number of program participants, distribution of participants by health care profession and State, and and total number of CME/CE/CEU certificates awarded; this information will inform AHRQ’s efforts to disseminate comparative effectiveness research information to clinician’s via online continuing education.
Participants’ Assessments of Educational Activity Effectiveness
Ratings of faculty effectiveness, scientific rigor, program objectivity, and the extent to which learning objectives were achieved; this information is collected immediately upon the participant’s completion of the CME/CE/CEU activity and will enable AHRQ to ascertain if the participant thinks that the learning methods utilized were effective.
Confidence and Attitudinal Change
Pre-activity versus post-activity measures of learners’ (a) confidence in using the comparative effectiveness research information in practice; and (b) attitude of the participant regarding the value of comparative effectiveness research in educating patients and making treatment decisions;. This information will indicate if the learning activity influenced confidence and made a difference in the participant’s attitude.
PerformanceMeasuring the extent that clinicians utilized comparative effectiveness research information to educate, counsel, and guide treatment decisions with their patients will help AHRQ evaluate if participants apply the knowledge received from the CME/CE/CEU activity.
AHRQ does not seek this exemption.
List of Attachments:
Attachment A – Healthcare Research and Quality Act of 1999
Attachment C – AHRQ Online Continuing Education Participant Evaluation
Attachment D – Federal Register Notice
Attachment E – Public Comments
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | DHHS |
| File Modified | 2011-07-28 |
| File Created | 2011-07-27 |