SUPPORTING STATEMENT
Part A
Online Submission Form for Supplemental Evidence and Data for Systematic Reviews for the Evidence-based Practice Center Program
Version: (December 9, 2015)
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
A. Justification 3
1. Circumstances that make the collection of information necessary 3
2. Purpose and use of information 5
3. Use of Improved Information Technology 6
4. Efforts to Identify Duplication 7
5. Involvement of Small Entities 8
6. Consequences if Information Collected Less Frequently 8
7. Special Circumstances 8
8. Consultation outside the Agency 8
9. Payments/Gifts to Respondents 9
10. Assurance of Confidentiality 9
11. Questions of a Sensitive Nature 9
12. Estimates of Annualized Burden Hours and Costs 9
13. Estimates of Annualized Respondent Capital and Maintenance Costs 10
14. Estimates of Annualized Cost to the Government 10
15. Changes in Hour Burden 11
16. Time Schedule, Publication and Analysis Plans 11
17. Exemption for Display of Expiration Date 12
List of Attachments 12
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote healthcare quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care;
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve healthcare quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special healthcare needs, including individuals with disabilities and individuals who need chronic care or end-of-life healthcare.
This is a new activity of AHRQ’s Evidence-based Practice Center (EPC) Program.
AHRQ’s EPC Program develops evidence reports and technology assessments on topics relevant to clinical and other health care organization and delivery issues—specifically those that are common, expensive, and/or significant for the Medicare and Medicaid populations. For example recent reviews have focused on clinical conditions, such as “Treatment of Nonmetastatic Muscle-Invasive Bladder Cancer”1; health delivery topics such as “Management Strategies to Reduce Psychiatric Admissions”2; and specific technologies such as “Imaging Techniques for Treatment Evaluation for Metastatic Breast Cancer.”3 These evidence reports include systematic reviews and technical briefs, and provide an essential foundation from which to understand what we know from existing research and what critical research gaps remain. These reports, reviews, and technology assessments are based on rigorous, comprehensive syntheses and analyses of the scientific literature on topics. EPC reports and assessments emphasize explicit and detailed documentation of methods, rationale, and assumptions. EPC reports are conducted in accordance with an established policy on financial and nonfinancial interests. These scientific syntheses may include meta-analyses and cost analyses.
The EPC Program supports AHRQ’s mission by synthesizing and disseminating the available research as a “science partner” with private and public organizations in their efforts to improve the quality, effectiveness, and appropriateness of health care. The EPC Program is a trusted source of rigorous, comprehensive, and unbiased evidence reviews for stakeholders. The resulting evidence reports and technology assessments are used by Federal and State agencies, private-sector professional societies, health delivery systems, providers, payers, and others committed to evidence-based health care. These end-users may use EPC Program evidence reports to inform policy decisions, clinical practice guidelines, and other healthcare decisions.
This research has the following goals:
Use research methods to gather knowledge on the effectiveness of certain treatments for specific medical conditions, both published and unpublished, to evaluate the quality of research studies and the evidence from these studies.
Promote the use of evidence in healthcare decision making to improve healthcare and health
Identify research gaps to inform future research investments
The Institute of Medicine standards for quality systematic reviews include an assessment of publication bias through the identification of unpublished studies. This is an important source for bias which could affect the nature and direction of research findings. Identifying and including the results of these additional unpublished studies may provide a more complete and accurate assessment of an intervention’s effect on outcomes. An important way to identify unpublished studies is through requests to medical device manufacturers, pharmaceutical companies, and other intervention developers.
The proposed project involves sending a request letter to relevant medical device manufacturers, pharmaceutical companies and other intervention developers to invite them to submit unpublished studies or other scientific information to the EPC Program website, with one request per systematic review topic. Because research on each topic must be completed in a timely manner in order for it to be useful, the collections are never ongoing—there is one request and collection per topic. Investigators in the EPC Program will review the information and assess potential risk of bias from both published and unpublished studies and its impact on the EPC Program’s findings. AHRQ believes the display of these assessments in the systematic review’s evidence tables will improve the response and submission rates of industry stakeholders by informing the healthcare community of the impact of potential bias on the research conclusions, and for healthcare decision making.
To achieve the goals of this project the following data collections will be implemented:
Online Submission Form Instrument. This information is collected for the purposes of providing supplemental evidence and data for systematic reviews (SEADS). The online submission form (OSF) collects data from respondents on their organization name, their product’s name, and whether they are providing all information on requested studies characteristic of the review in progress. This happens following receipt of a request letter from the SRC. These requests will be sent to relevant sponsors of preventative and treatment interventions (e.g., medical device manufacturers, pharmaceuticals, and other intervention and health care system developers), with one request per topic. For the purposes of meta-analyses, trial summary data from missing and unidentified studies are sought. For the purposes of constructing evidence tables and quality ratings (e.g. on public reporting of cost measures or health information exchange), data can vary (e.g., URLs, study designs, and consumer-mediated exchange forms). Information on both completed and ongoing studies are requested.
The EPC Program, through the SRC, currently uses a Federal Register notice and broad- based email announcement to stakeholders to allow the public to know about each topic, and the opportunity to submit scientific information. In 2014, the Program sent 517 notifications to 336 industry stakeholders. Of those 517 announcements sent, 14.1% received a response; 56.2% of the responses (or 7.9% of all requests) contained submissions of information on the results of interventions. This experience has prompted this proposed project.
The additional use of direct requests to relevant organizations would improve the Program’s ability to obtain this information. Contacting intervention sponsors for missing and potentially unidentified studies could improve the impact of research efforts and downstream dissemination efforts, and could positively impact the health of individuals (along with their supporting communities) burdened by poor health. Including information about response data to these requests to more accurately characterize the completeness of the evidence in the systematic reviews may also address this issue.
The proposed project does not duplicate other available sources of this information. Available study registries and databases may not be complete to sufficiently inform the Program’s research.
The purpose of SEADS requests is not to collect generalizable data, but to supplement the published and grey literature searches EPC investigators are conducting. Furthermore, considering the evidence and data included in responses collected from industry stakeholders, an assessment pertaining to the completeness of the evidence-base will be produced. This, AHRQ believes, will increase the value of AHRQ’s research reviews to end-users and potentially provide stakeholders a better understanding of how their submissions are used.
The EPC Program, through the SRC, currently uses a Federal Register notice and broad- based email announcement to stakeholders to allow the public to know about each topic, and the opportunity to submit scientific information. The additional use of direct requests (Appendix A) to relevant organizations would improve the Program’s ability to obtain this information. The SRC plans to conduct one SEADS collection per topic. Twenty-four topics per year with SEADS requests are anticipated, with an average range of 10 to 30 potential respondents per topic. The EPC Program does not anticipate more than 40 topics per year with SEADS requests.
The Effective Health Care website houses information and documents specific to the EPC Program. Through this website, documents are shared with the public, and give stakeholders the opportunity to comment on interim documents, such as the proposed scope of a product and a draft report. The Effective Health Care website would also serve as a gateway for the electronic submission of information and materials (SEADS), allowing access to an online submission form (OSF; see Attachment B for an outline) upon the finalization the research scope for the individual topics. Users of the SRC-controlled OSF website will be industry stakeholders and investigators involved in the sponsoring of studies on interventions and healthcare strategies related to the topics investigated by the Program. The responses and submissions are intended to be included in statistical analyses and general assessments related to the completeness of the evidence-base used to evaluate the different treatment options for patients suffering from the conditions under study.
The information can be uploaded as a MS Word document, PDF, or as a ZIP file, which potentially reduces the burden on the submitter. A portal will be open for at least four weeks for each topic. If the interventions under study include devices or other intervention types not requiring the ingestion of any substances, this period will coincide with the Federal Register Notice. The OSF is not a questionnaire.
From a range of fields concerning the submitter and their information, there will be only three required fields in the OSF in addition to any files they wish to upload. The required fields are the submitter’s organization, the intervention, and a statement about the completeness of their submission. Submitters may choose to include additional details, such as their preferred contact information (e.g., e-mail, phone, address). Consequently, in the online submission form submitters will be informed that the preferred format is the completion of the entire form.
The optional file (Data Entry Form) is available as an alternative to the OSF. Responders may download and use it to submit information. This form summarizes the content of the SEADS. Before respondents gain entry to the data entry portion of the OSF, they will have the option to download an excel file and fill out information related to studies they have sponsored. Respondents may upload the completed file when uploading specific study information.
In addition to electronic submission of SEADS through the Effective Health Care Program website, respondents may also e-mail the SEADS Coordinator their files directly or send materials through the mail.
The OSF includes details about what type of information would be most helpful to the EPC Program. It states that this is a voluntary submission. Submitters are informed that the contents of all submissions will be made available to the public upon request. All SEADS are reviewed by the SRC SEADS Coordinator and the EPC investigator team.
Through AHRQ and the SRC, the EPC Program currently uses a Federal Register notice to allow the public to know about ongoing topics and the opportunity to submit scientific information. The use of direct requests to relevant organizations would improve the Program’s ability to obtain this information. While the Program has worked with representatives from the Food and Drug Administration (FDA) when part of a stakeholder panel, and attempted to obtain publicly available information from relevant FDA resources, because the information submitted to the FDA is proprietary information, it may be heavily redacted and limit its usefulness. Moreover, the Electronic Freedom of Information Act (eFOIA) of 1996 means that FDA materials like drug approval packages are readily available only after 1996. Thus, a standard FOIA is required for those studies completed up to 1996. However, FOIA request are described on FOIA.gov to take about a month for simple requests and much longer for more complicated requests. Since the systematic reviews conducted by EPCs are on a short schedule to ensure their prompt use in healthcare settings, additional time for FOIAs are likely not practical.
Additional factors limiting the usefulness of FDA resources are that the FDA only conducts approvals for pre-marketing studies with specific labeling most reliably available for primary efficacy outcomes. This leaves out information on post-marketing studies, off-label uses, and many secondary efficacy outcomes. For these data, ClinicalTrials.gov is an important resource. However, it is only recently that results are required to be uploaded in addition to the trials being registered on ClinicalTrials.gov. Furthermore, studies subject to regulation by the FDA, such as investigational device exemptions, are not required to be registered on ClinicalTrials.gov; and if these studies fail regulatory testing, such as futility analyses, the FDA will not make their outcomes or circumstances available to the public on their website since the device has likely not been approved.
The passing of Section 801 of the Food and Drug Administration Amendments Act (FDAAA 801) in September of 2007 means that results of trials conducted before this date are not required to be posted on ClinicalTrials.gov. Thus, identified trials on ClinicalTrials.gov older than this date without results would likely require FOIAs as well and, in reference to the statement two paragraphs above, this is not a highly viable option due to time constraints.
This activity does not intend to intentionally involve nor exclude or impact any small entities. The process used to collect data is designed to minimize the burden on all respondents. The OSF for SEADS includes three required fields and allows for the submission of any scientific material. These fields are the organization name, intervention name, and declaration of the completeness of the submission. This is the minimum required information.
This is a one-time collection for each topic. If this collection is not conducted, it will negatively impact the scientific rigor and comprehensiveness of the research. Moreover, this research is intended to inform clinician and patient decisionmaking in healthcare, and guidance in clinical practice. An incomplete assessment of the evidence due to the absence of runs the risk of biasing these decisions, and negatively impacting health outcomes for individuals and future research investments by researchers and research funders.
A particular manufacturer may develop an intervention that is used for multiple topics, or related topics. If this arises an effort will be made to check previous submissions on related topics.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on September 21, 2015 on page 56989 for 60 days (see Attachment D).
The SRC will consult with outside consultants on general and specific areas of the OSF. The consultants the SRC has identified are:
Harlan Krumholz, MD (Yale School of Medicine);
Kay Dickersin, PhD (Johns Hopkins Bloomberg School of Public Health); and
Steven Goodman, MD, PhD (Stanford School of Medicine).
No payments or gifts to respondents will be given.
Information that can directly identify the respondent, such as a person’s name and/or social security number will not be collected. Section 944 (c) of the Public Health Service Act [42 U.S.C. 299c 3(c)]requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied unless they consent to the use of the information for another purpose.
This activity does not entail questions of a sensitive nature.
Exhibit 1 presents estimates of the reporting burden hours for the data collection efforts. Time estimates are based on pilot testing of materials and what can reasonably be requested of respondents. The number of respondents listed in “Number of respondents per SEADS request” of Exhibit 1 reflects a projected 80% response rate.
Online Submission Form: A form for submitting scientific evidence and data related to medical interventions sponsored by organizations and individuals such as pharmaceutical companies and independent researchers. The form has three required fields: the organization’s name, the intervention in question, and whether the information they provide is all the information they know to exist. They may upload documents and they are also provided a data entry form (Attachment C) if they wish to offer greater details on their studies.
An Optional Data Entry Form (Attachment C) is available as an alternative to the Online Submission form.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents per SEADS request |
Number of responses per respondent |
Hours per response |
Total burden hours per SEADS |
Online Submission Form (OSF) |
70 |
1 |
15/60 |
17.5 |
Total |
70 |
1 |
15/60 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of SEADS requests |
Total burden hours per SEADS |
Average hourly wage rate* |
Total cost burden |
OSF |
70 |
17.5 |
$55.48a |
$970.90 |
Total |
70 |
17.5 |
$55.48 |
$970.90 |
aBased on the mean wages for Public Relations and Fundraising Managers, 11-2031, the occupational group most likely tasked with completing the OSF.
There are no direct costs to respondents other than their time to participate in the study.
The total cost of this data collection to the government is $12,564 per year; $10,500 in contract costs and $2,064 in government personnel costs. The data collection is a one-time collection per topic. Exhibit 3 shows a breakdown of the total cost and annualized cost for the data collection and data processing and analysis led by the contractor. Exhibit 4 shows a breakdown of the government personnel costs related to this data collection effort.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
NA |
NA |
Data Collection Activities |
$21,000 |
$7000 |
Data Processing and Analysis |
$3000 |
$1000 |
Publication of Results |
NA |
NA |
Project Management |
$6000 |
$2000 |
Overhead |
1500 |
500 |
Total |
$31,500 |
$10,500 |
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours per topic |
Number of topics per year |
Cost |
Data Collection Oversight |
GS-14 |
$51.60 |
0.5 |
40 |
$1,368 |
Total |
|
|
|||
Annual salaries based on 2015 OPM Pay Schedule for Washington/DC area: http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2015/DCB.pdf
Each SEADS request is a new collection. The nature of systematic reviews is to secure comparable evidence on the efficacy and effectiveness of numerous treatments for health related diseases and disorders. These reviews aim to inform healthcare decisionmaking by clinicians and consumers, and inform guidance on clinical practice. The findings of these reviews are intended to help clinicians and consumers make the best decisions in their particular circumstances. In general, the goal for these reports is to be completed within a year. The steps that go into each review therefore are on a tight schedule and are not ongoing in order to fulfill their purpose. Thus, there are no ongoing collections of information from study sponsors and industry stakeholders for the same topic.
Exhibit 5 Idealized Data Collection Timeline for Each SEADS
|
|
Description (in chronological order) |
Due Date
|
Request/receive list of intervention sponsors (contacts) from EPC investigators |
Roughly 1 month following EPC award date of systematic review |
Create database records for list of contacts for the purposes of mailing |
Before final protocol posting |
Final protocol of research review |
Roughly two months after contact list received |
Open SEADS submission portal |
Within 3 days of final protocol |
Send SEADS request letters |
Concurrent with portal opening |
Send submission deadline reminder (e-mail) |
2 weeks after letters sent |
Close SEADS submission portal |
4 weeks after letters sent |
Alert EPC investigator team of portal closure |
Within 2 days of portal closure |
Data analysis |
4-8 months after portal closure |
Final report (AHRQ publication) |
7-9 months after portal closure |
Publication Plan:
Research review results will be
disseminated through a peer-reviewed publication under the auspices
of the AHRQ EPC and EHC Programs.
Analysis
Plan:
Provided any data submitted by intervention sponsors is not redundant and is useful for the purposes of either meta-analysis or evidence tables, the EPC investigator team will include it in the research review. In addition to this use of the data, a table will be created illustrating the intervention sponsor’s response to the SEADS request.
Exhibit 6. SEADS Collection and Analysis Plans
Instrument |
When administered and to whom |
Analysis sub-goal |
Analysis plan |
SEADS Request Letter (Attachment A) |
|
None |
None |
Online Submission Form (Attachment B) |
|
Tabulate the responses and non-responses from sponsors to assess their impact on the systematic review. |
|
Data Entry Form (Not Required) |
|
Tabulate the responses and non-responses from sponsors to assess their impact on the systematic review. |
|
AHRQ does not seek this exemption.
List of Attachments:
Attachment A -- Supplemental Evidence and Data for Systematic Reviews Request Letter
Attachment B -- Website portal for Submission of Supplemental Evidence and Data for Systematic Reviews
Attachment C – Optional Data Entry Form
Attachment D -- Federal Register Notice
1 Chou R, Selph S, Buckley D, Gustafson K, Griffin J, Grusing S, Gore J. Treatment of Nonmetastatic Muscle-Invasive Bladder Cancer. Comparative Effectiveness Review No. 152. (Prepared by the Pacific Northwest Evidence-based Practice Center under Contract No. 290-2012-00014-1.) AHRQ Publication No. 15-EHC015-EF. Rockville, MD: Agency for Healthcare Research and Quality; June 2015.
2 Gaynes BN, Brown C, Lux LJ, Ashok M, Coker-Schwimmer E, Hoffman
V, Sheitman B, Viswanathan M. Management Strategies To Reduce Psychiatric Readmissions.
Technical Brief No. 21. (Prepared by the RTI-UNC Evidence-based Practice Center under
Contract No. 290-2012-00008-I.) AHRQ Publication No.15-EHC018-EF. Rockville, MD:
Agency for Healthcare Research and Quality. May 2015.
www.effectivehealthcare.ahrq.gov/reports/final.cfm.
3 Gold LS, Lee CI, Devine B, Nelson H, Chou R, Ramsey S, Sullivan SD.
Imaging Techniques for Treatment Evaluation for Metastatic Breast Cancer. Technical Brief No.
17. (Prepared by the Pacific Northwest Evidence-based Practice Center under Contract No. 290
2012-00014-I.) AHRQ Publication No. 14-EHC044-EF. Rockville, MD: Agency for Healthcare
Research and Quality; October 2014. www.effectivehealthcare.ahrq.gov/reports/final.cfm.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement Part A Template |
| Subject | <p>(Supporting Statement Template and Instructions; instructions are in italics and should be deleted when complete) |
| Author | Heather Nalls |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |