SUPPORTING STATEMENT
Part A
Continuing Education for Comparative Effectiveness Research Survey
Version: (July 8th, 2014)
Agency for Healthcare Research and Quality (AHRQ)
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 4
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 6
6. Consequences if Information is Collected Less Frequently 6
7. Special Circumstances 6
8. Consultation Outside the Agency 6
9. Payments/Gifts to Respondents 6
10. Assurance of Confidentiality 7
11. Questions of a Sensitive Nature 7
12. Estimates of Annualized Burden Hours and Costs 7
13. Estimates of Annualized Respondent Capital and Maintenance Costs 8
14. Estimates of Annualized Cost to the Government 8
15. Changes in Hour Burden 9
16. Time Schedule, Publication and Analysis Plans 9
17. Exemption for Display of Expiration Date 10
List of Attachments 10
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care.
2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators.
3. Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Patient-centered outcomes research (PCOR) is an area that has seen increased focus from research agencies and other government entities. Also known as comparative effectiveness research, PCOR is the focus of one of AHRQ’s research portfolios, which has the mission of providing health care decision-makers (e.g., patients, healthcare providers, purchasers, and policymakers) with recent evidence-based information about the harms, benefits, and effectiveness of various treatment options. The purpose of PCOR is to inform health care decisions without providing clinical recommendations by comparing medical devices, surgeries, tests, drugs, or ways to deliver health care.
Section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 created AHRQ’s Effective Health Care (EHC) Program, which became the first federal program to conduct PCOR and disseminate those findings to the public. AHRQ works with researchers, academic organizations, and research centers through the EHC program to produce research focused on effectiveness and comparative effectiveness across a variety of clinical areas and translate that research into products for a variety of stakeholders to help spread awareness and knowledge about PCOR. It is important for AHRQ to be able to measure the effectiveness of these products, which include training modules and publications, specifically around how they are affecting health care professionals’ understanding, awareness, and use of PCOR and its related concepts. The Affordable Care Act of 2010 cast new light and emphasis on the importance of PCOR, and it is important for AHRQ to be able to gauge the effectiveness of its current materials and identify ways to improve how this information is being disseminated to the medical community.
The Continuing Education for Comparative Effectiveness Research Project is designed to provide online continuing education materials that inform physicians and other healthcare providers about patient-centered health research from the Effective Health Care Program, specifically comparative effectiveness research reports, and other similar reports from AHRQ-approved sources. Online multimedia continuing education modules based on the Effective Health Care Program comparative effectiveness research reports will be planned, developed, disseminated, and promoted. In addition, data will be collected on the modules to assess their effectiveness and impact.
To achieve the goals of this project the following data collections will be implemented:
Each participant will complete a pre-module assessment to determine their baseline understanding and knowledge around the topics covered in the module. Anticipated respondents to this data collection will include nurses, nurse practitioners, physicians, physician assistants, pharmacists, health education specialists, case managers, dieticians, psychologists, medical assistants, social workers, and other allied health professionals.
Immediately upon completing the module, learners will also complete:
A program evaluation assessing the training in terms of meeting objectives, faculty performance and ways to improve future training events.
A post-module assessment to capture the change in comprehension around comparative effectiveness research that has occurred as a result of the module.
Each training module will also involve one follow-up questionnaire that is administered six months after the completion of the course for the purposes of tracking the longer-term effectiveness of the modules.
Simple inferential tests (e.g. paired t-tests) will be employed to examine changes over time between questions asked pre-module, post-module, and 6-months post-module.
This study is being conducted by AHRQ through its contractor, Hayes Inc. (Hayes) and Hayes’ subcontractors, Deloitte Consulting LLP (Deloitte), pursuant to AHRQ’s statutory authority to support the agency’s dissemination of comparative clinical effectiveness research findings. 42 U.S.C. 299b-37(a)-(c).
2.1: Purpose of the Data Collection
This data collection will help to meet AHRQ’s objectives to:
Understand the extent to which these online continuing education modules based on the Effective Health Care Program comparative effectiveness research reports improve knowledge of each topic and change participants’ awareness of, attitude towards, and/or confidence to apply CER in their clinical practice.
Track information about the dissemination efforts employed for CE/CER information specific to the modules, and the uptake of AHRQ’s other Effective Health Care Program materials as a result of the project, including the Clinician and Consumer Summaries when available.
Determine implementation practices (e.g. changes in practice behavior or implementation of the information conveyed in the modules) that occur as a result of the learning.
Identify opportunities for improving the presentation and delivery of CE modules by gathering information on the participants’ reactions to the modules and to the faculty presenters through the post-event evaluation assessment.
2.2: Who Will Use the Information
AHRQ and the EHC Program staff will use the information collected through this Information Collection Request to assess the short- and long-term progress in achieving the dissemination and implementation aims of the Continuing Education project
The survey will be administered as a web-based survey. AHRQ will use contractor, Hayes Inc. (Hayes), and subcontractor, Deloitte Consulting LLC (Deloitte), to manage the online data collection and development of the Continuing Education for Comparative Effectiveness Research Survey web interface.
The six-month follow-up survey data will be gathered through a Learning Management System as part of each learning activity. Utilizing the Learning Management System, Deloitte will send a digitally-signed email to each learner six months after he/she has completed the CE activity. The email will include a link (URL) to the web-based survey and provide a text explanation of the study to help encourage learners to participate. The respondents will be able to complete the survey using their personal computers, tablet, or other web-enabled mobile device. All responses will be kept confidential. All responses will be stored on computer servers with software and hardware solutions that have a high degree of resistance to tampering and circumvention, including both hardware and software firewalls.
In designing the survey, available information on each subject was carefully reviewed to ensure that the survey gathers information not available from existing sources. The current request is intended to support a new training program for the Agency and, therefore, the instruments are specific and unique to this project. Similar data was collected from a previous continuing education contract which resulted in over 62,000 continuing education certificates being issued to health care professionals The data collected from the previous continuing education contract is being used to inform the present project. This new data collection is for a new set of educational modules that will be released beginning June 2014.
No small businesses or other small entities will be involved as respondents in the proposed data collection effort. Respondents to this data collection will be nurses, nurse practitioners, physicians, physician assistants, pharmacists, health education specialists, case managers, dieticians, psychologists, medical assistants, social workers, and other allied health professionals. Efforts have been made to minimize response burden on respondents through careful design of the data collection strategy and efficient construction of the data collection instrument.
Each training module will involve one follow-up questionnaire that is administered six months after the completion of the course for the purposes of tracking the longer-term effectiveness of the modules. Gathering this information less frequently would prohibit AHRQ from being able to determine the impact of these modules on variables like dissemination of information and application of CER information in practice. This data cannot be captured in the post-module evaluation as participants would not yet have had a chance to act on the information they have learned. Administering the survey after more than six months runs the risk of being too long after the module, and will affect respondents’ abilities to recall instances where they have applied their learning.
There are no special circumstances that require deviation from the general information collection guidelines of 5 CFR 1320.5(d)(2)..
8.a. Federal Register Notice
In accordance with 5 CFR 1320.8(d), this information collection soliciting public comments was announced in the Federal Register on August 4, 2014 (Volume 79 Number149, Page 45192) for 60 days (Attachment C). AHRQ’s contractor, Hayes Inc. worked with its subcontractor, Deloitte, to develop the data collection instrument and data collection methodology in consultation with staff from AHRQ.
All respondents are being asked to complete the survey voluntarily. No payments will be provided for participation.
The surveys will also include the OMB control number, expiration date, and the Public Burden Statement. This document specifies the Authority supporting the request for information, the purpose for its collection, the routine uses to which it will be put, the scope of anonymity in the use of personal identifiers and the voluntary nature of participation.
Email addresses provided by health care professionals at the time of taking the online educational activity will be used to contact learners for the survey described in this supporting statement. The contractor involved in this survey, Hayes Inc., has signed a confidentiality agreement, assuring that any contact information and information provided by respondents will be used solely for the purposes of this survey. Hayes is bound to keep this type of information confidential by its contract with AHRQ and the contract contains a data use agreement, which establishes AHRQ as the “owner” of this data. There will be an option at the end of the survey for learners to provide additional contact information if they are willing to be contacted by AHRQ about their experience.
Survey participants will be notified via digitally signed email that contains a link to the posttest assessment, and describes that their information will be kept confidential, will not be released to third parties other than AHRQ, and will not be reported except in an aggregated manner.
The information collected will be handled using procedures designed to keep sensitive personal information secure. Electronic data files will be password protected and any files transferred will be encrypted and password protected. Upon completion of the fieldwork, data is extracted and processed using SPSS software. The finished SPSS (Statistical Package for Social Sciences) data file will not contain any identifiable information.
The survey does not contain questions that are considered sensitive in nature. Participants will be given the option to provide their contact information to enable follow up on their experience with the CME / CE and how it impacted their practice, but this question will not ask for anything beyond name and contact information (e.g., email address).
Exhibit 1 provides information on the estimated time to complete the data collection survey. These educational activities are enduring training modules and will be available for a 2-year period. The survey will be administered to each individual 6 months after completing the training. As many as 4,400 health care professionals are expected to complete the survey, based on an average of 2,000 health care providers taking each module with a 10% response rate, or 200; 200 x 22 modules = 4,400. On average, respondents will spend 5 minutes completing the survey. Therefore the total burden for data collection for the survey is estimated at .08 x (# respondents) hours.
Exhibit 1: Estimated Respondent Burden
A |
B |
C |
D |
E |
F |
G |
Estimated Number of Respondents |
Average Burden per Respondent (Minutes) |
Total Annual Burden (Minutes) |
Number of Responses per Respondent |
Total Respondent Burden (Minutes) |
Total Burden per Respondent (Minutes) |
Total Respondent Burden (Hours) |
|
|
(A*B) |
|
(C*D) |
(B*D) |
(E/60) |
4400 |
5 |
22,000 |
1 |
22,000 |
5 |
367 |
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
AHRQ Online CME/CE 6-Month Evaluation |
4,400 |
367 |
$49.83 |
$18,288 |
Total |
4,400 |
367 |
N/A |
$18,288 |
*Based upon the mean of the average hourly wages for Physicians (29-1069; $92.25), Pharmacists (29-1051; $56.01), Physician Assistants (29-1071; $45.36), Nurse Practitioners (29-1171; $45.71), Registered Nurses (29-1111; $33.13), and Healthcare Practitioners (29-9099; $26.54), May 2013 National Occupational Employment and Wage Estimates, United States, U.S. Department of Labor, Bureau of Labor Statistics. http://www.bls.gov/oes/current/oes_nat.htm#29-0000 viewed May 5, 2014.
There is no cost to respondents, other than the time required to respond to the survey.
The total contracted cost to the federal government for the Continuing Education for Comparative Effectiveness Research Surveys to be administered analyzed and reported under this contract is $276,242. This includes costs to develop, program and pre-test the surveys, administer the surveys, provide survey responses and costs to analyze survey responses and develop and produce the accompanying 30-day, 6-month, 18-month and 2-year reports. These costs have been calculated over a three-year period (36 months total) and are assumed to be consistent across each year.
Exhibit 3a. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$ 34,724.00 |
$ 11,574.67 |
Data Collection Activities |
$ 116,811.00 |
$ 38,937.00 |
Data Processing and Analysis |
$ 51,535.00 |
$ 17,178.33 |
Publication of Results |
$ 19,724.00 |
$ 6,574.67 |
Project Management |
$ 29,086.00 |
$ 1,650.00 |
Overhead |
$ 24,362.00 |
$ 9,695.33 |
Total |
$ 276,242.00 |
$ 92,080.67 |
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours |
Cost |
Data Collection Oversight |
Health Communications Specialist |
Grade13/Step 10 $56.01 |
50 |
$2,800.50 |
Review of Results |
Health Communications Specialist |
Grade13/Step 10 $56.01 |
40 |
$2,240.40 |
Total |
$5,040.90 |
|||
http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2014/general-schedule/
This submission to OMB is a new request for approval; there is no change in burden.
Contingent upon OMB approval, this survey is scheduled to be administered from December 2014 through December 2017. The primary objective of the analysis of the data collected will be to evaluate the effectiveness of the training modules with respect to knowledge gained, changes in attitudes and confidence regarding CER, application of CER knowledge, and dissemination efforts.
Tabulations will be conducted for multiple questions via a frequency analysis to measure effectiveness of the modules among nurses, nurse practitioners, physicians, physician assistants, pharmacists, health education specialists, case managers, dieticians, psychologists, medical assistants, social workers, and other allied health professionals. The majority of data will be descriptive data and will be presented in the form of frequency tables. Simple inferential tests (e.g. paired t-tests) will be employed to examine changes over time between questions asked pre-module, post-module, and 6-months post-module. Should parametric test assumptions not be met, it is also possible to create a categorical variable representing percent change in score (e.g., 0%-25% change; 26%-50% change, 51%-75% change, 76%-100% change) and perform a chi-square goodness-of-fit test to examine the extent to which different background characteristics are distributed as expected across the parameter change categories. Qualitative responses to open-ended questions related to participants’ reactions to the training and experience applying the learning in practice will be analyzed for common themes. Subject to the quantity of qualitative data, themes will either be analyzed manually or uploaded into AtlasTI for thematic content analysis.
AHRQ does not seek this exemption.
List of Attachments:
Attachment A – “Measuring Attitudes on AHRQ’s Continuing Education Resources:
A Survey after 6 Months”
Attachment B – “6 Month Follow Up Survey Email”
Attachment C -- Federal Register Notice
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| File Modified | 0000-00-00 |
| File Created | 2021-01-26 |