SUPPORTING STATEMENT A for
Process Evaluation of the NIH Roadmap Epigenomics Program (NIDA)
May 2011
Social & Scientific Systems, Inc. (SSS) with funding from the National Institute on Drug Abuse (NIDA)
NIDA Project Officer:
Genevieve deAlmeida-Morris, Ph.D., MPH.,
Office of Science Policy and Communications
NSC - Neuro Science Center, 5229
6001 Executive Blvd
Rockville, MD 20852
Phone: 301-594-6802
Fax: 301-480-2485
E-mail: [email protected]
1. Circumstances Making the Collection of Information Necessary 3
2. Purpose and Use of the Information Collection 7
b. Survey Content and Method of Administration 9
3. Use of Information Technology and Burden Reduction 11
4. Efforts to Identify Duplication and Use of Similar Information 12
5. Impact on Small Businesses or Other Small Entities 12
6. Consequences of Collecting the Information Less Frequently 13
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 13
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 13
9. Explanation of Any Payment or Gift to Respondents 14
10. Assurance of Confidentiality Provided to Respondents 15
11. Justification for Sensitive Questions 16
12. Estimates of Hour Burden Including Annualized Hourly Costs 16
a. Estimated Annualized Burden Hours 16
b. Annualized Cost to Respondents 17
13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers 17
14. Annualized Cost to the Federal Government 18
15. Explanation for Program Changes or Adjustments 18
16. Plans for Tabulation and Publication and Project Time Schedule 18
17. Reason(s) Display of OMB Expiration Date is Inappropriate 19
18. Exceptions to Certification for Paperwork Reduction Act Submissions 20
ATTACHMENTS for Supporting Statement A:
Attachment A-1: Roadmap Epigenomics Program: RFA Requirements for Evaluation
Attachment A-2: References
Attachment A-3: Principal Investigator Survey
Attachment A-4: IRB Approval letter
Process Evaluation of the NIH Roadmap Epigenomics Program (NIDA)
SUPPORTING STATEMENT
Part A
The proposed information collection is essential to the process evaluation of the National Institutes of Health (NIH) Roadmap Epigenomics Program. The evaluation is part of the funding requirement of the program, and participation in it is required of each awardee of the program components as stated in the Requests for Applications (RFA) of this NIH Roadmap Epigenomics Program, as per the below excerpt from the RFA for Technology Development in Epigenetics (R01), RFA-RM-07-011:
“As part of good program management, NIH assesses the implementation and effectiveness of its programs using evaluation tools and techniques. Grantees may be asked to provide information for program evaluation purposes, both locally and at the national level. Such information may be used in evaluations of the Technology Development projects, as well as the “Mid-Course” review of the entire Roadmap Epigenomics Program.” (Part II, Section I.1, Evaluation)
This requirement is addressed more specifically in some of the RFAs, for example, in the Epigenomics Data Analysis and Coordination Center RFA – EDACC (U01), RFA-RM-07-014, Section VI.3, Reporting- there is an additional requirement that states:
“Periodically, throughout the life of the program, awardees may be required to provide data that can be used to evaluate program progress, such as (but not limited to) …..Collaboration with other Roadmap Epigenomics Program participants (REMCs, NCBI, Technology RFA awardees, demonstration project RFA awardees)”
A full list of the evaluation requirements in the RFAs funded under the Roadmap Epigenomics Program is shown In Attachment A-1, Roadmap Epigenomics Program RFA Requirements for Evaluation.
The information collection will survey principal investigators (PI) receiving grants under the NIH Roadmap Epigenomics Program about the mechanisms of research coordination and collaboration and the results and synergies from these interactions as part of the Roadmap Epigenomics Program. This information is methodologically critical because it will augment and expand the information available from secondary data sources and tracking systems for the process evaluation of this program. This is a one-time information collection from the Principal Investigators receiving NIH funding under the NIH Roadmap Epigenomics Program.
The NIH Roadmap Epigenomics Program, one of the NIH Common Fund’s “cross-cutting, exceptionally high-impact, trans-NIH programs,” supports NIH’s mission “to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce the burdens of illness and disability.” (http://www.nih.gov/about/mission.htm). The goals of the NIH Roadmap Epigenomics program (EP) are: to establish multiple sets of comprehensive reference epigenomes; develop new reagents and tools for epigenetic research; identify public resources for purified high quality stem cells, differentiated cells, and tissues; provide publicly accessible data as well as new tools for data integration; conduct research to identify and characterize novel epigenetic marks; and conduct research to develop and test novel hypotheses on epigenetic roles in human health and disease, stimulate the development of revolutionary epigenetic technologies that will significantly change the way that epigenetics research is performed.
The NIH Roadmap Epigenomics Program comprises a series of complementary initiatives aimed at generating new research tools, technologies, datasets, and infrastructure to accelerate understanding of the role of epigenetics - the study of how chemical "marks" on DNA regulate gene activity and expression without altering the DNA sequence itself - in human health and disease. The program’s five components are:
Reference Epigenome Mapping Centers
Epigenomics Data Analysis and Coordination Center
Technology Development in Epigenetics
Discovery of Novel Epigenetic Marks in Mammalian Cells
Epigenomics of Human Health and Disease
The hypothesis of this NIH Roadmap Program is that the coordinated and synergistic program components and their interventions will lead to the program goals outlined above. The phenomenon of this multiple-component research program is complex and has required a unique approach to the program evaluation.
The Epigenomics Roadmap Program has planned for and included various evaluation activities over the Roadmap Program’s 10-year period, the process evaluation among them. This evaluation study to assess the program process and progress is non-experimental. The assessment is primarily on secondary source information, with primary source information collection added to augment the reliability and internal validity. The primary data collection uses information categories that genuinely tap added distinctions and opinions that relate to it to build the weight of evidence from first-hand sources that substantiate the initial hypotheses about the program phenomenon and its differences from a typical research portfolio of individual and insular projects.
This request is for Office of Management and Budget (OMB) approval, under the Paperwork Reduction Act of 1995, for a one-year generic clearance for the NIH to conduct process evaluation, including a survey of the principal investigators of the research teams, of the NIH Roadmap Epigenomics Program. Authorization which makes this information collection necessary is in in U.S.C. Title 42, Chapter 6A, Subchapter III, Part A, Subpart 282 (Director of National Institutes of Health), Section (b) Duties and authority:
“(4) shall assemble accurate data to be used to assess research priorities, including information to better evaluate scientific opportunity, public health burdens, and progress in reducing minority and other health disparities;
(5) Shall ensure that scientifically based strategic planning is implemented in support of research priorities as determined by the agencies of the National Institutes of Health
The process evaluation is funded from the Evaluation Set-Aside Program under the Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI) within the Office of the Director of NIH, through a funding award to the National Institute on Drug Abuse (NIDA). This process evaluation of the program was preceded by an evaluability assessment; an outcome evaluation is also planned for conduct after year five of the Program.
The synthesized results across primary and secondary data sources will provide critical insights on transformativeness of high-impact, trans-NIH programs and contribute important information about the synergies and collaborations in multi-component scientific research. It will also identify areas for program improvement and decision-making at the Program’s midpoint so that NIH program staff could refine the focus of the program to ensure meeting program goals, and to inform planning for the future of the program. Results will contribute to better informed decision-making and use of resources along with lessons learned that could be useful to the Agency’s programs.
This information collection clearance request is for the collection of primary source data, a one-time Principal Investigator survey, as part of the process evaluation of the NIH Roadmap Epigenomics Program; this information will augment data from the Program’s secondary sources – the progress reports and tracking and information systems. Results from this information collection on the mechanisms of research coordination and collaboration and the results that have had synergistic and transformative effects from these interactions under the Roadmap Epigenomics Program will provide NIH with important information about the complementary and possibly magnifying effects of teams of researchers conducting research and sharing results under a unified program vision and conceptual framework. Directly surveying PIs will also provide NIH with early evidence of global use of research results - dissemination and diffusion of the findings and results of the research under this Program beyond the grantees funded - well in advance of formal publications by researchers outside of this Program.
The purpose of the process evaluation, a formative evaluation study which is descriptive in nature, is to examine the implementation and progress of the program, up to mid-course in the program. The evaluation findings will report the achievement from the implementation and process of a multiple-component research program, will contribute lessons for developing an innovative program that requires coordination and collaboration across multiple components, will identify the collaboration and synergies of a coordinated data processing and analysis center and four individual reference mapping centers providing validated epigenome reference maps, and describe evidence of global use of the data and research results. It may also contribute lessons for evaluating the process of an NIH Roadmap Program. In addition, the results will provide the counterfactual evidence substantiating the concept of this Roadmap Program – that it would be greatly more contributing than the way epigenomics research was being conducted by independent and unrelated teams of researchers. Following a review of the NIH portfolio and the state of the science globally, NIH and experts in the field convened at an NIH-sponsored Technical Workshop 2007 and identified that the existing research paradigm was not producing the scientific research results needed to improve the health.
This process evaluation began in mid-2009 and is scheduled to be completed by November 2011, in time to report on the program at mid-course. The evaluation research is a multi-method design to take advantage of the strengths of both primary and secondary data sources. Although program records and available monitoring and information-systems data will provide documentation to assess what has been accomplished, primary data sources will illuminate how the work has been accomplished. Of particular interest are the patterns of interactions among grantees that facilitate or inhibit a progression toward scientific advances and the structures and processes that have supported integration of disciplines and projects. The survey of principal investigators will expand and explain further the information gained from the secondary document review and data systems analysis. (Additional primary information may be collected from in-depth interviews with nine or fewer PIs.1 These interviews will be conducted after the PI survey results are in, to explore topics such as collaboration mechanisms for which more information is needed.)
The survey contains several conceptual topic areas: Overall Epigenomics Program Synergies and Opportunities, Productivity and Efficiency, Innovations, Access to and Use of Epigenomics Program Resources, Research Progress, Collaboration, among others). The survey instrument includes adaptations from instruments such as that developed for the Transdisciplinary Tobacco Use Research Centers (TTURC) Initiative2 to enable assessment of the transdisciplinary collaboration3 of the NIH Roadmap Epigenomics Program research teams. (See Attachment A-2 for References.)
The PI survey will be conducted using an online survey to reduce respondent burden, thus improving the survey response rate, and to improve data quality (since data entry from paper-based forms is not required). Respondents will be able to access the survey from the convenience of their office, laboratory, or home computers, and this will also improve response rates. NIH Program staff will send an introductory email to Participants informing them about the PI survey and that they will be contacted by SSS who will be conducting the survey. Participants will then receive from SSS an email request (including a URL link to the survey with a study ID along with survey directions) for their participation in the survey, with collateral materials explaining the survey’s purpose and use of the information collection, the non-personal nature of the information including protection by blinding, and analyses to be conducted in the aggregate only. All PIs will be asked to complete the survey within two (2) weeks.
Online data collection enables rapid data analysis and reporting of results and increases data quality. Response rates will be maximized, with the goal of achieving as close to 100% response rate as possible, by the use of automated notification reminders to participants at two intervals after the initial survey notification is emailed to them. This schedule is shown in Table A.2.b-1 below.
Table A.2.b-1 Participant Recruitment Contacts |
|
Contact Item |
Timing (from time of first contact) |
Introductory email from NIH Program staff |
Day 1 |
Follow-up email from SSS survey liaison with survey URL, password |
Days 1-2 |
First reminder email asking for completion within 5 days |
Day 14 |
Second reminder email asking for completion within 3 days |
Day 21 |
Phone contact for any PIs who haven’t completed survey |
Day 28-29 |
The information collection (survey) instrument will be pretested by nine or fewer individuals from among NIDA, NIEHS, and NIDDK staff, and senior research staff from the evaluation Contractor. This pretest will be conducted in the first quarter of calendar year 2011.
Information technology will be used to collect and process information to reduce the burden on the public. The survey instrument in Attachment A-3 will be administered in a web-based format to reduce respondent burden. In addition, the electronic survey complements the very highly-technical nature of this scientific program and the highly technical PIs who are conducting the research of this Roadmap Epigenomics Program. Respondents will be able to access the survey from the convenience of their office, laboratory, or home computers which will further reduce respondent burden.
This evaluation study is a mixed-methods study which primarily uses secondary source documentation and information from tracking and monitoring systems; it cannot, however, be conducted without the primary source information collection being requested here to augment the reliability and internal validity of the process evaluation. After conducting a comprehensive assessment of the available Roadmap Epigenomics Program data sources, the Contractor Senior Staff for this study recommended that primary data needs to be collected in addition to the secondary data sources available, to augment and clarify the reported facts and effectively conduct the process evaluation. Since the Roadmap Epigenomics Program is a Common Fund research program and is thus a unique program, there is no other evaluation of similar research programs that would provide information needed in this evaluation. All information from this evaluation will be reported directly to the funding Agency.
The collection of information under consideration in this supporting statement does not include small businesses. This evaluation is funded by NIDA from an award from NIH Evaluation Set-Aside funds. Participants are principal investigations from U.S. academic institutions that are not small businesses and their participation is a funding requirement (as described in Section A.1. above).
This request is for a one-time survey as part of the required process evaluation for the available Roadmap Epigenomics Program, at the mid-course of the five-year program. As described above, it will contribute greatly to NIH’s understanding of the conduct of research among and across synergistic teams, enabling NIH to consider more efficient and/or productive funding mechanisms to achieve larger research impact. If this information is not collected, NIH will not have a complete perspective on the Program’s progress after three years of implementation nor will NIH be able to describe and/or advocate for similar programs which would have the ability to provide research results – and health impact - in a more rapid and/or more efficient fashion.
This request is consistent and complies with the general information collection guidelines of 5 CFR 1320.5. No special circumstances apply.
As required by 5CFR 1320.8, notices of this proposed data collection were submitted and published as shown in Table A.8-1. No comments were received to the 60-day notice.
Federal Register Notice |
Submission Date |
Federal Register Publication Date |
Publication Location |
Comments Received? (Yes, No) |
60-day |
March 8, 2011 |
March 14, 2011 |
Vol. 76, No. 49, pg. 13648-13649 |
No |
Two internationally renowned, academic experts are part of the Contractor team for this evaluation. Dr. John D. Roessner, an evaluation researcher and Professor Emeritus, School of Public Policy, Georgia Institute of Technology, is the Academic Advisor. Dr. Bernard Weissman, an epigenomics researcher at the UNC Lineberger Comprehensive Cancer Center of the University of North Carolina School of Medicine, is the Epigenomics Expert. They serve as expert Advisors on methodological considerations in evaluating a funded research program and application of evaluation to the nature of the science of the program. They developed this information collection instrument along with the Contractor Evaluation Senior Staff for the study. The survey was then shared with the NIH Project Officer and the other NIH evaluation leads for their review and comments after which NIH Program staff also reviewed and provided input.
There are no payments or gifts to respondents. We anticipate that the response rate will be 100% because this process evaluation, of which this information collection (the PI survey) is part, is a required condition of each grantee’s funding.
The proposed research has been reviewed and approved as exempt by the Institutional Review Board that SSS uses. (Confirmation letter of Human Subjects Institutional Review Board review and approval is shown in Attachment A-4.) The information provided will be kept confidential and will not be disclosed to anyone but the researchers conducting the study, except as otherwise required by law. Confidentiality during data preparation and analysis will be maintained by using subject ID numbers rather than names on the data collection form. (When matching selected respondent forms with respondent names provided initially, for the interview group, only the Contractor staff for the evaluation will have access to the personally identifiable information.) Any identifying information inadvertently provided by participants will be promptly removed. Online data will be maintained on a secure server during the duration of the research. Data linking the subject ID with identifiers and all printed records will be kept in a locked filing cabinet accessible only to the evaluation experts and project assistants involved in this evaluation research.
No personally identifiable information is being requested in this survey; during analysis, data will be cleaned in such a way that any information which might allow identification of the respondent (from an open-ended response) will be deidentified. In addition, results will be aggregated for all reports or any published material, in such a way that no Personally Identifiable Information can be obtained. All electronic and paper data pertaining to identifiable participant data will be securely stored for three years, in keeping with NIH requirements, and then purged, unless otherwise directed by NIDA. Standard human subjects guidelines will be followed.
"Cookies" will not be employed as part of any online survey mechanism. A cookie is a small file that a Web site transfers to a user's hard disk to allow the Web server to record specific information about the user's session while they are visiting the Web site. As the participants will be voluntarily supplying requested information, there is no need to surreptitiously collect other details behind the scenes.
No questions of a sensitive nature are included in the survey.
Based on the advice of the expert advisors, contractor’s previous experience in web survey administration, and in web-based pre-testing with fewer than 10 respondents, it is estimated that participants will require no more than 20 minutes to complete the PI survey. (Actual time required will vary based on participant reading speed and level.) The annual burden table below (Table A.12.a-1.) shows the total annual burden to participants for this one-time survey.
Table A.12.a-1. Estimated Annualized Burden Hours |
|||||
Type of Respondent |
Number of Respondents |
Number of Responses per Respondent |
Annual Number of Responses |
Avg. Burden Hours Per Response |
Annual Burden Hours Requested |
Principal Investigators |
53 |
1 |
1 |
0.33 (20 minutes) |
17.49 |
Based on the planned respondents (principal investigators), the average hourly rates shown in Table A.12.b-1 are the average of the 50th percentile (median) and top 10th percentile of hourly wages of the category, Medical Scientists, Except Epidemiologists. The wage estimates are derived from the National Occupational Employment and Wage Estimates, Bureau of Labor Statistics, May 2009.
Table A.12.b-1. Estimated Annualized Burden Costs |
||
Annual Burden Hours Requested |
Average Hourly Wage Rate* |
Total Annual Respondent Cost |
17.49 |
$51 |
$891.99 |
There are no direct costs to respondents other than their time to participate in the study. There will be no additional cost to the respondent for capital equipment, software, computer services, or maintenance to provide the information required by this research. They will use internet technology already available to them and will access the survey website at no cost to their research grants from the Roadmap Epigenomics Program.
The approximate total cost to the government for this study is $249,099. It is estimated that the equivalent of 10%-time of one NIH staff member will be required to devote 200 hours for the duration of the proposed research. Assuming an annual salary of $100,000 total government personnel costs will be $10,000. Evaluation contractor costs are $239,099. This price includes costs for research design, pretesting, development of the Office of Management and Budget clearance package, data collection, data entry, data analyses, and development of a final report.
This is a new data collection activity.
This information collection will require a maximum of 4 months (expected: June – September 2011). The timeline provided in Table A.16-1 is based upon the assumption that OMB clearance for the information collection is received by May 2011.
Table A.16-1 Project Time Schedule |
|
Activity |
Time Schedule |
Pretest the Epigenomics Program Principal Investigator Survey |
Pretest is in process while awaiting OMB approval |
Implement Principal Investigator Survey |
Within 2 weeks after OMB approval |
Conduct analyses |
2 - 3 months after OMB approval |
Present results at NIH-sponsored meeting |
4 months after OMB approval |
Since this information collection (survey) comprises part of a process evaluation which is formative and descriptive in nature, analyses will be descriptive; there are no survey hypotheses. The results from the process evaluation, including the results from this information collection, will be presented at an NIH-sponsored meeting and summarized in a final printed report for limited and internal NIH distribution.
Process evaluation results (in the aggregate) will be reported to the NIDA Project Officer as required of the Contractor.
No exemption is requested. The OMB expiration date will be displayed on the information collection instrument as shown in the Principal Investigator Survey shown in Attachment A-3:Principal Investigator Survey.
There are no exceptions to the certification statement identified in OMB Form 83-I, item 19, “Certification for Paperwork Reduction Act Submissions.”
1 not subject to OMB Review under Paperwork Reduction Act of 1980
2 Masse LC, Moser RP, Stokols D, Taylor BK, Marcus SE, Morgon GD, Hall KL, Croyle RT and Trochim WM. Measuring collaboration and transdisciplinary integration in team science. Am J Prev Med 2008;35(2S):S151-160.
3 Stokols D, Hall KL, Taylor BK, and Moser RP. The Science of Team Science: Overview of the Field and Introduction to the Supplement. Am J Prev Med 2008;35(2S):S77–S89)
Feb. 7, 2010: EP PI
Survey-OMB Supporting Statement A
| File Type | application/msword |
| Author | staffer coda |
| Last Modified By | curriem |
| File Modified | 2011-05-18 |
| File Created | 2011-05-18 |