SUPPORTING STATEMENT
Part A
Online Submission Form for Supplemental Evidence and Data for Systematic Reviews for the Evidence-based Practice Center Program
Version: (June 28, 2022)
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 5
3. Use of Improved Information Technology 6
4. Efforts to Identify Duplication 7
5. Involvement of Small Entities 8
6. Consequences if Information Collected Less Frequently 8
7. Special Circumstances 8
8. Consultation outside the Agency 8
9. Payments/Gifts to Respondents 9
10. Assurance of Confidentiality 9
11. Questions of a Sensitive Nature 9
12. Estimates of Annualized Burden Hours and Costs 9
13. Estimates of Annualized Respondent Capital and Maintenance Costs 10
14. Estimates of Annualized Cost to the Government 10
15. Changes in Hour Burden 11
16. Time Schedule, Publication and Analysis Plans 11
17. Exemption for Display of Expiration Date 12
List of Attachments 12
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote healthcare quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care;
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve healthcare quality.
This is an ongoing activity of AHRQ’s Evidence-based Practice Center (EPC) Program.
AHRQ’s EPC Program develops evidence reports and technology assessments on topics relevant to clinical and other health care organization and delivery issues—specifically those that are common, expensive, and/or significant for the Medicare and Medicaid populations. For example recent reviews have focused on clinical conditions, such as “Radiation Therapy for Brain Metastases”1; health delivery topics such as “Transitions of Care From Pediatric to Adult Services for Children With Special Healthcare Needs”2; and specific technologies such as “Telehealth for Women’s Preventive Services.”3 These evidence reports include systematic reviews, technical briefs, and rapid reviews; and provide an essential foundation from which to understand what we know from existing research and what critical research gaps remain. These reports, reviews, and technology assessments are based on rigorous, comprehensive syntheses and analyses of the scientific literature on topics. EPC reports and assessments emphasize explicit and detailed documentation of methods, rationale, and assumptions. EPC reports are conducted in accordance with an established policy on financial and nonfinancial interests. These scientific syntheses may include meta-analyses and cost analyses.
The EPC Program supports AHRQ’s mission by synthesizing and disseminating the available research as a “science partner” with private and public organizations in their efforts to improve the quality, effectiveness, and appropriateness of health care. The EPC Program is a trusted source of rigorous, comprehensive, and unbiased evidence reviews for stakeholders. The resulting evidence reports and technology assessments are used by Federal and State agencies, private-sector professional societies, health delivery systems, providers, payers, and others committed to evidence-based health care. These end-users may use EPC Program evidence reports to inform policy decisions, clinical practice guidelines, and other healthcare decisions.
This research has the following goals:
Use research methods to gather knowledge on the effectiveness and harms of certain treatments and healthcare delivery processes and models for medical conditions, both published and unpublished, to evaluate the quality of research studies and the evidence from these studies.
Promote the use of evidence in healthcare decision making to improve healthcare and health
The Institute of Medicine (now National Academies of Sciences, Engineering, and Medicine) standards for quality systematic reviews include an assessment of publication bias through the identification of unpublished studies. This is an important source for bias which could affect the nature and direction of research findings. Identifying and including the results of these additional unpublished studies may provide a more complete and accurate assessment of an intervention’s effect on outcomes. An important way to identify unpublished studies and data is through providing an opportunity for the public to share this information, including medical device manufacturers, pharmaceutical companies, and other intervention developers .
The proposed project involves a broad-based stakeholder email and an additional posting a notice in the federal register for selected review topics to reach relevant medical device manufacturers, pharmaceutical companies and other intervention developers and increase awareness of the opportunity to submit unpublished studies or other scientific information to the EPC Program website, with one portal per systematic review topic. Because research on each topic must be completed in a timely manner in order for it to be useful, the collections are never ongoing—there is one portal and collection per topic. Investigators in the EPC Program will review the information, and note whether additional studies from the SEADS submissions will be included in the review. They will assess potential risk of bias from both published and unpublished studies if they are relevant to the review, and include in the analysis. AHRQ believes this is needed for transparency and to maintain rigor of the evidence review. In addition it may improve the response and submission rates of industry stakeholders by informing the healthcare community of the impact of potential bias on the research conclusions, and for healthcare decision making.
To achieve the goals of this project the following data collection will be implemented:
Online Submission Form. This information is collected for the purposes of providing supplemental evidence and data for systematic reviews (SEADS). The online submission form (OSF) collects data from respondents on their name, organization name, description of the submission, medical condition, intervention, and email address. . For the purposes of meta-analyses, trial summary data from missing and unidentified studies are sought. For the purposes of constructing evidence tables and quality ratings (e.g. on public reporting of cost measures or health information exchange), data can vary (e.g., URLs, study designs, and consumer-mediated exchange forms). Information on both completed and ongoing studies are requested. Submitters may alternatively email their submission to the AHRQ EPC mailbox at [email protected].
The EPC Program currently uses broad- based email announcement to stakeholders and in some cases an additional a Federal Register notice, to allow the public to know about each topic, and the opportunity to submit scientific information. In 2021, the Program opened 11 SEADS portals on the Effective Health Care website and 10 had a corresponding Federal Register notice. 73% (8/11) of all SEADS portals received a response; and all received research material considered for inclusion in the review. Eight of the 10 SEADS portals with a corresponding Federal Register notice had responses. This experience has prompted continuation of this proposed project.
The proposed project does not duplicate other available sources of this information. Available study registries and databases may not be complete to sufficiently inform the Program’s research.
This study is being conducted by AHRQ pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
The purpose of SEADS submissions is not to collect generalizable data, but to supplement the published and grey literature searches EPC investigators are conducting. Furthermore, considering the evidence and data included in responses collected from industry stakeholders, an assessment pertaining to the completeness of the evidence-base will be produced. This, AHRQ believes, will increase the value of AHRQ’s research reviews to end-users and potentially provide stakeholders a better understanding of how their submissions are used.
The EPC Program currently uses broad- based email announcement to stakeholders and through AHRQ listserves, and in some cases an additional Federal Register notice to allow the public to know about each topic, and the opportunity to submit scientific information. AHRQ plans to conduct one SEADS collection per topic. Up to twenty-four topics per year with SEADS portals are anticipated; over the past 5 years the number of SEADS portals has ranged from 11-20; with an average range of 0-5 potential respondents per topic. The EPC Program does not anticipate more than 40 topics per year with SEADS portals.
The Effective Health Care website houses information and documents specific to the EPC Program. Through this website, documents are shared with the public, and give stakeholders the opportunity to comment on interim documents, such as the proposed scope of a product and a draft report. The Effective Health Care website would also serve as a gateway for the electronic submission of information and materials (SEADS), allowing access to an online submission form (OSF; see Attachment B) upon the finalization the research scope for the individual topics. Submitters using the SEADS OSF will be the public, including industry stakeholders and investigators involved in the sponsoring of studies on interventions and healthcare strategies related to the topics investigated by the Program. The responses and submissions are intended to be included in statistical analyses used to evaluate the different treatment options or healthcare processes for patients .
The information can be uploaded as a MS Word document, PDF, Excel, or as a ZIP file, which potentially reduces the burden on the submitter. A portal will be open for at least four weeks for each topic. If the interventions under study include devices, generic drugs, or poorly defined interventions (such as non-drug, health delivery interventions) a Federal Register notice will be posted and this period will coincide with the Federal Register Notice. The OSF is not a questionnaire.
From a range of fields concerning the submitter and their information, there will be only one required field in the OSF in addition to any files they wish to upload. The required field is the submitter’s name. Submitters may choose to include additional details, such as their e-mail, organization name. The submitter may provide an email address to receive acknowledgement of their submission.
In addition to electronic submission of SEADS through the Effective Health Care Program website, respondents may also e-mail the EPC Program their files directly.
Guidance for the OSF includes details about what type of information would be most helpful to the EPC Program. It states that this is a voluntary submission. Submitters are informed that the contents of all submissions will be made available to the public upon request. All SEADS are reviewed by AHRQ and the EPC investigator team.
The EPC Program currently uses a broad-based stakeholder email and Federal Register notice to allow the public to know about ongoing topics and the opportunity to submit scientific information. While the Program has worked with representatives from the Food and Drug Administration (FDA) when part of a stakeholder panel, and attempted to obtain publicly available information from relevant FDA resources, because the information submitted to the FDA is proprietary information, it may be heavily redacted and limit its usefulness. Moreover, the Electronic Freedom of Information Act (eFOIA) of 1996 means that FDA materials like drug approval packages are readily available only after 1996. Thus, a standard FOIA is required for those studies completed up to 1996. However, FOIA request are described on FOIA.gov to take about a month for simple requests and much longer for more complicated requests. Since the systematic reviews conducted by EPCs are on a short schedule to ensure their prompt use in healthcare settings, additional time for FOIAs are likely not practical.
Additional factors limiting the usefulness of FDA resources are that the FDA only conducts approvals for pre-marketing studies with specific labeling most reliably available for primary efficacy outcomes. This leaves out information on post-marketing studies, off-label uses, and many secondary efficacy outcomes. For these data, ClinicalTrials.gov is an important resource. However, it is only recently that results are required to be uploaded in addition to the trials being registered on ClinicalTrials.gov. Furthermore, studies subject to regulation by the FDA, such as investigational device exemptions, are not required to be registered on ClinicalTrials.gov; and if these studies fail regulatory testing, such as futility analyses, the FDA will not make their outcomes or circumstances available to the public on their website since the device has likely not been approved.
The passing of Section 801 of the Food and Drug Administration Amendments Act (FDAAA 801) in September of 2007 means that results of trials conducted before this date are not required to be posted on ClinicalTrials.gov. Thus, identified trials on ClinicalTrials.gov older than this date without results would likely require FOIAs as well and, in reference to the statement two paragraphs above, this is not a highly viable option due to time constraints.
This activity does not intend to intentionally involve nor exclude or impact any small entities. The process used to collect data is designed to minimize the burden on all respondents. The OSF for SEADS includes one required field in addition to the submission of any scientific material. These field is the name. This is the minimum required information.
This is a one-time collection for each topic. If this collection is not conducted, it will negatively impact the scientific rigor and comprehensiveness of the research. Moreover, this research is intended to inform clinician and patient decision-making in healthcare, guidance in clinical practice, and healthcare policy. An incomplete assessment of the evidence due to the absence of runs the risk of biasing these decisions, and negatively impacting health outcomes for individuals and future research investments by researchers and research funders.
A particular manufacturer may develop an intervention that is used for multiple topics, or related topics. If this arises an effort will be made to check previous submissions on related topics.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on July 19, 2022 on page 43039 Vol. 87, No. 137 for 60 days (see Attachment C). AHRQ received no comments from the public.
AHRQ will consult with outside consultants on general and specific areas of the OSF. The consultants AHRQ has identified are:
Edwin Reid(Portland VA Research Foundation)
Amanda Borsky, PhD (Veterans Health Administration)
Jimmy Le, Sc.D. (NIH/National Eye Institute)
No payments or gifts to respondents will be given.
The OSF will collect a person’s name; other information such as a person’s social security number will not be collected. Section 944 (c) of the Public Health Service Act [42 U.S.C. 299c 3(c)]requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied unless they consent to the use of the information for another purpose. Information about the submitter will not be shared publicly.
This activity does not entail questions of a sensitive nature.
Exhibit 1 presents estimates of the reporting burden hours for the data collection efforts. Time estimates are based on pilot testing of materials and what can reasonably be requested of respondents. The number of respondents listed in “Number of respondents” of Exhibit 1 reflects a projected upper range response rate per SEADS portal multiplied by the anticipated upper limit of number of SEADS portals per year, based on historical information over the past 3 years.
Online Submission Form: A form for submitting scientific evidence and data related to medical interventions sponsored by organizations and individuals such as pharmaceutical companies and independent researchers. The form has one required field in addition to uploading a document: the submitter’s name.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours per SEADS |
Online Submission Form (OSF) |
200 |
1 |
15/60 |
50 |
Total |
200 |
1 |
15/60 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Respondent Cost |
OSF |
200 |
50 |
$57.62a |
$2,881 |
Total |
200 |
50 |
$57.62 |
aBased on the mean wages for Public Relations and Fundraising Managers, 11-2030, the occupational group most likely tasked with completing the OSF.
There are no direct costs to respondents other than their time to participate in the study.
The total cost of this data collection to the government is $19,444per year; $16,000 in contract costs and $3,444in government personnel costs. The data collection is a one-time collection per topic. Exhibit 3 shows a breakdown of the total cost and annualized cost for the data collection by the website contractor. Exhibit 4 shows a breakdown of the government personnel costs related to this data collection effort.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost (over 3 years) |
Annualized Cost (assuming 40 SEAD/year) |
Project Development |
NA |
NA |
Data Collection Activities |
$48,000 |
$16,000 |
Data Processing and Analysis |
NA |
NA |
Publication of Results |
NA |
NA |
Project Management |
NA |
NA |
|
|
|
Total |
$48,000 |
$16,000 |
1
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours per topic |
Number of topics per year |
Cost |
Review of SEADS submission |
GS-14 |
$60.49 |
1.0 |
40 |
$2,420 |
Data Collection Oversight |
GS-13 |
$51.18 |
0.5 |
40 |
$1,024 |
Total |
|
$3,444 |
|||
Annual salaries based on 2022 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2022/DCB_h.pdf/
Each SEADS portal is a new collection. The nature of systematic reviews is to secure comparable evidence on the efficacy and effectiveness of numerous treatments for health related diseases and disorders. These reviews aim to inform healthcare decision-making by clinicians and consumers, and inform guidance on clinical practice. The findings of these reviews are intended to help clinicians and consumers make the best decisions in their particular circumstances. In general, the goal for these reports is to be completed within a year. The steps that go into each review therefore are on a tight schedule and are not ongoing in order to fulfill their purpose. Thus, there are no ongoing collections of information from study sponsors and industry stakeholders for the same topic.
Exhibit 5 Idealized Data Collection Timeline for Each SEADS
|
|
Description (in chronological order) |
Due Date
|
Final protocol of research review posted on EHC website |
|
Open SEADS submission portal |
Within 3 days of final protocol |
Send broad-based email |
Concurrent with portal opening |
Post Federal Register notice (if needed) |
Within 2 weeks of portal opening |
Close SEADS submission portal |
4 weeks after SEADS submission portal opened |
Alert EPC investigator team of portal closure |
Within 2 days of portal closure |
Data analysis |
4-8 months after portal closure |
Final report (AHRQ publication) |
7-9 months after portal closure |
Publication Plan:
Research review results will be
disseminated through a peer-reviewed publication under the auspices
of the AHRQ EPC Program.
Analysis
Plan:
Provided any data submitted by intervention sponsors is not redundant and is useful for the purposes of either meta-analysis or evidence tables, the EPC investigator team will include it in the research review.
Exhibit 6. SEADS Collection and Analysis Plans
Instrument |
When administered and to whom |
Analysis sub-goal |
Analysis plan |
SEADS Email (Attachment A) |
|
None |
None |
Online Submission Form (Attachment B) |
|
Tabulate the number of responses |
|
AHRQ does not seek this exemption.
List of Attachments:
Attachment A -- Supplemental Evidence and Data for Systematic Reviews Email
Attachment B -- Website portal for Submission of Supplemental Evidence and Data for Systematic Reviews
Attachment C -- Federal Register Notice
Attachment E – Draft PIA
11 Garsa A, Jang JK, Baxi S, Chen C, Akinniranye O, Hall O, Larkin J, Motala A, Newberry S, Hempel S. Radiation Therapy for Brain Metastases. Comparative Effectiveness Review No. 242. (Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-2015-00001-I.) AHRQ Publication No. 21-EHC021. PCORI Publication No. 2020-SR-02. Rockville, MD: Agency for Healthcare Research and Quality; June 2021. DOI: 10.23970/AHRQEPCCER242
2 Parsons HM, Abdi HI, Nelson VA, Claussen A, Wagner BL, Sadak KT, Scal PB, Wilt TJ, Butler M. Transitions of Care From Pediatric to Adult Services for Children With Special Healthcare Needs. Comparative Effectiveness Review No. 255. (Prepared by the Minnesota Evidence-based Practice Center under Contract No. 75Q80120D00008.) AHRQ Publication No. 22-EHC027. Rockville, MD: Agency for Healthcare Research and Quality; May 2022. DOI: https://doi.org/10.23970/AHRQEPCCER255.
3 Cantor A, Nelson HD, Pappas M, Atchison C, Hatch B, Huguet N, Flynn B, McDonagh M. Effectiveness of Telehealth for Women’s Preventive Services. Comparative Effectiveness Review No. 256. (Prepared by the Pacific Northwest Evidence-based Practice Center under Contract No. 75Q80120D00006.) AHRQ Publication No. 22-EHC024. Rockville, MD: Agency for Healthcare Research and Quality; June 2022. DOI: https://doi.org/10.23970/AHRQEPCCER256.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement Part A Template |
| Subject | <p>(Supporting Statement Template and Instructions; instructions are in italics and should be deleted when complete) |
| Author | Heather Nalls |
| File Modified | 0000-00-00 |
| File Created | 2022-10-21 |