Supporting Statement A For:
Surveys and Interviews to Support an Evaluation of the Innovative Molecular Analysis Technologies (IMAT) Program (NCI)
February 20, 2015
Tony Dickherber
Center for Strategic Scientific Initiatives
National Cancer Institute
31 Center Dr., Rm 10A33
Bethesda, MD 20892
Telephone: 301.547.9980
Fax: 301.480.2889
E-mail: [email protected]
Table of Contents
A. Justification iii
A.1 Circumstances Making the Collection of Information Necessary 1
A.2. Purpose and Use of the Information Collection 2
A.3 Use of Improved Information Technology and Burden Reduction 4
A.4 Efforts to Identify Duplication and Use of Similar Information 5
A.5 Impact on Small Businesses or Other Small Entities 5
A.6 Consequences of Collecting the Information Less Frequently 5
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 6
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 6
A.9 Explanation of Any Payment or Gift to Respondents 6
A.10 Assurance of Confidentiality Provided to Respondents 6
A.11 Justification for Sensitive Questions 7
A.12 Estimates of Annualized Burden Hours And Costs 7
A.13 Estimates of Other Total Annual Cost Burden to Respondents and
Record Keepers 9
A.14 Annualized Cost to the Federal Government 9
A.15 Explanation for Program Changes or Adjustments 10
A.16 Plans for Tabulation and Publication and Project Time Schedule 10
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 12
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 12
List of Attachments
Attachment 1: IMAT Awardee Interview Guide
Attachment 2: IMAT Awardees and Other NIH Awardees Evaluation Web-based Survey
Attachment 3: IMAT Technology End User Interview Guide
Attachment 4: Background and Rationale
Attachment 5: Trans-NIH Evaluation Advisory Committee (EAC)
Attachment 6: Privacy Act Memo
Attachment 7: Office of Human Subjects Research Protection (OHSRP) Exemption
Attachment 8: Invitation Letters to IMAT Awardees, Technology End Users, and IMAT Awardees and Other NIH Awardees (Comparison Group)
ABSTRACT
This request is to gain OMB approval for the new submission titled, “Surveys and Interviews to Support an Evaluation of the Innovative Molecular Analysis Technologies (IMAT) Program” for 1 year. The National Cancer Institute (NCI) IMAT program was launched in 1998 to support the development of highly innovative technologies to advance cancer research and clinical care capabilities. NCI is proposing to pursue a comprehensive and robust evaluation to assess the process and outcomes of the IMAT program, and also seek opportunities for NCI to improve the program’s utility for the broad continuum of cancer researchers, clinicians and ultimately patients. The focus of the evaluation will be on understanding the successes of supported technologies and will occur with three respondent groups: IMAT awardees, a sample of awardees from NIH funded programs on technology development beyond the IMAT program, and technology end users. The evaluation approach is to be centered on tracking or following all supported technologies since 1998. To date, the IMAT program has issued 513 R21 and R33 awards and 160 SBIR and STTR awards, supporting roughly 500 unique technology platforms.
A.1 Circumstances Making the Collection of Information Necessary
The National Cancer Institute (NCI) Innovative Molecular Analysis Technologies (IMAT) program was launched in FY1998. The mission for the IMAT program is “to support the development, maturation, and dissemination of novel and potentially transformative next-generation technologies through an approach of balanced but targeted innovation in support of clinical, laboratory, and epidemiological research on cancer”. Given the trans-divisional nature of the program, it has always been managed from the NCI Office of the Director. This is consistent with the purpose of NCI and authorizes the collection of information under Section 410 of the Public Health Service Act (42 USC § 285).
While the structure of the IMAT Program has evolved over periodic reformulations of the associated funding opportunities, the goal remains largely unchanged since its inception which is to support highly innovative ideas focused on early-stage technology development unlikely to win support through traditional NIH funding mechanisms. The IMAT program generally includes a rolling portfolio of 60 to 90 active projects, with total award outlays (including new and continuing awards) of approximately $20M to $30M per year. The program also requires participation at an annual meeting from all funded investigators (Attachment 4).
A.2 Purpose and Use of the Information Collection
NCI is pursuing a comprehensive process and outcome assessment of the 15-year old IMAT program. While the program consistently offers promising indicators of success, the full program has not been evaluated since 2008, and never in as comprehensive a manner as has been formulated in the current evaluation plan. An outcome evaluation of the long-standing NCI IMAT program presents a rich and unique opportunity likely to serve institutes across the NIH, and perhaps other federal agencies, considering the costs and benefits of directing resources towards supporting technology development. An award through the NIH Evaluation Set-Aside program to support this evaluation, for which NIH-wide relevance is a principle element of determining merit for support, is testament to this. The evaluation serves as an opportunity to gauge the impact of investments in technology development and also to assess the strengths and weaknesses of phased innovation award mechanisms.
The focus of the evaluation will be on understanding the successes of supported technologies, specifically. The program's longevity provides a large data set with the potential to produce statistically significant findings and also comes with a variety of highly leverage-able resources, including: currently active program staff with extensive institutional memory; a robust feasibility study with proposed outcome evaluation design; and partial evaluation studies of the program to reduce the scope of work for the proposed evaluation.
The evaluation approach is to be centered on tracking or following all supported technologies since 1998. The history of the technology would then be tracked so that the technology can be described at each stage of its development, even prior to conceptualization for IMAT funding. Interviews and surveys will be conducted with individuals who have different associations with the technology in order to gather information about the current state of the technology and its potential for affecting progress in cancer research and treatment.
The interviews and surveys will occur with three different respondent groups including: IMAT awardees, a comparison group of NIH awardees, and technology end-users, possibly including patient advocacy communities. Technology end-users are respondents who employed any technologies that arose from IMAT funding, but were personally not involved during the IMAT-supported periods of development1. All three groups of respondents comprise primarily of scientists, engineers, and clinicians. Interview and survey protocols will focus on collecting the following information:
Identification of all IMAT and associated technologies:
What were the pre-existing technologies that served as the basis for technology developed by IMAT?
What kinds of technologies were proposed and what types were funded?
Identify the development path(s) for IMAT technologies:
How were the technologies developed during the funding period?
What is the development path after initial funding?
How is the technology developed after IMAT funding and/or disassociation of the original PI with the project?
Dissemination of all IMAT technologies:
How were the details of the technology spread to scientific audiences?
To what extent is the technology or methodology being used?
Outcomes or impacts of successfully developed IMAT technology:
What are the short-term and intermediate-term impacts?
Assessment of IMAT Program Design:
How did the application process, solicitation, and IMAT funding structure (mechanisms) impact the development of your technology?
Assessment of IMAT Program Implementation:
How did interactions with NIH, NCI, or other organizations impact the development of your technology?
How did the research environment (e.g., institutional support; other related research activities) impact the development of your technology?
Interactions and collaborations with NCI program staff and other research organizations.
Experiences with IMAT during the application and award selection process.
Experiences with the funding mechanisms and the grants related to IMAT.
The web-based survey protocol (Attachment 2) complements the interview protocols (Attachments 1 and 3). Issuing the web-based survey to a comparison group of NIH supported investigators similarly developing novel technologies offers an important comparison group for the evaluation study. Responses to the surveys and interviews from IMAT awardees, who are the researchers that developed the technology, allow for the identification of research scientists in the field that might have employed these technologies, representing the Technology End Users.
Web-based surveys and telephone interviews will be employed to reduce the burden to the respondent. A database will be developed that contains the names of the respondents, the contact information, and the dates of all email and telephone contacts that will be used to track the most up-to-date contact information. All quantitative data will be standardized and compiled into a searchable database. The qualitative data will be compiled into Word documents and personally identifying information about the respondents from the database will be kept separate. If deemed necessary and worthwhile by the evaluation lead, more sophisticated data management and qualitative analysis software tools (e.g. nVivo software) may be employed.
The NCI Privacy Coordinator has been consulted and a determination as to whether a Privacy Impact Assessment (PIA) is needed is ongoing.
NCI conducted three previous pilot evaluations2 of this program in 2008, 2010 and 2013 to assess outcomes similar to the aims of the proposed evaluation, as they were based, in part, on the same professionally-developed evaluation design originally proposed in 2007 guiding the current evaluation strategy. Outcomes from these pilot evaluations were largely positive, suggesting significant anecdotal evidence that the IMAT program was achieving its proposed goals, but were unsatisfactory in that they represented such a narrow investigation of the full portfolio and highly susceptible to selection bias. An archival data analysis will be conducted to select particular respondents based on their case profile, however no subjective information exists about their experiences with the IMAT program or with individuals who are using technologies that arose from IMAT funding. This is the first comprehensive and robust evaluation conducted since the IMAT Program’s inception in 1998.
SBIR and STTR grantees that were awarded IMAT grants comprise about 25% (160) of the awards made. The investigators involved in these projects would be approximately 25 of the awardees selected to be interviewed. As for other interviews, telephone surveys will be employed to limit the burden on the interviewee.
This is a one-time collection.
This study complies fully with the guidelines of 5 CFR 1320.5. No exceptions to the guidelines are required.
A 60-day Federal Register Notice was published on December 4, 2014 (Vol. 79, P. 72004) and allowed 60 days for public comment and review. There were no public comments received.
A trans-NIH Evaluation Advisory Committee (EAC) was recruited to make early recommendations on the evaluation plan, consider ongoing progress of the evaluation, and to comment on the findings and final report from the proposed evaluation (Attachment 5).
No payment or gift will be made to the respondents.
A.10 Assurance of Confidentiality Provided to Respondents
Respondents will have the option to skip any question they would prefer not to answer and to quit the survey at any time. Unless express permission is provided by the respondent, all data will be de-identified and reported in the aggregate. Respondents will not be asked to complete a consent form. Each respondent’s willingness to schedule time for an interview will be interpreted as evidence of implied consent.
To protect the security of respondents’ information, all project files will be password protected and access to the files will be limited to authorized project staff. Interview information will be stored on a secure server protected with a Secure Sockets Layer (SSL) certificate and 128-bit encryption, the strongest online data encryption protection available. The tracking database with individual contact information will be stored separately from the data. The database will contain IDs only. The tracking database that links IDs to individual information will be destroyed at the end of the project. Project reports will not identify individuals who completed the survey. No names, university names, or personal identifying information will be used in any published reports of this study unless given express permission from the respondent. Survey reports will present all findings in aggregate so individual responses cannot be identified.
The NIH Privacy Act Officer has reviewed this data collection and determined the Privacy Act is applicable and is covered by NIH Privacy Act Systems of Record Notice (SORN) #09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” (Attachment 6). This SORN was published in Federal Register on 9/26/2002, Vol. 67, p. 60743.
Additionally, the Office of Human Subjects Research Protection (OHSRP) has reviewed this project and deemed that Federal regulations for the protection of human subjects do not apply to this information collection, and thus it has been excluded from Institutional Review Board review (Attachment 7).
The questions being asked do not constitute sensitive questions.
There will be a total of 950 individuals completing web-based surveys and interviews: 450 IMAT awardees and 450 other NIH grant awardees focused on technology development, and 50 technology end-users. The 450 IMAT awardees and 450 other NIH grant awardees (total of 900 respondents) will complete web-based surveys (Attachment 2), and the 50 technology end-users will complete the interview (Attachment 3).
Additionally, 100 respondents (of the IMAT awardees who responded to the survey) will be asked to participate in a follow-up telephone interview to develop a more sophisticated understanding of their development experience (Attachment 1).
It is estimated that the total and annualized burden will be 575 hours over the one-year information collection request (Table A.12-1). Additionally, Federal employees will be interviewed and asked similar questions, but because this is part of their job duties, the burden is not calculated into the table below.
Table A.12-1 Estimate of Annual Burden Hours
Form Name |
Type of Respondents |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per Response (in hours) |
Total Annual Burden (in hours) |
IMAT Grantee Interview
|
IMAT Awardees |
100 |
1 |
1 |
100 |
Technology Grantees Web-based Survey
|
IMAT Awardees; Other NIH Awardees representing comparison group |
900 |
1 |
30/60 |
450 |
Tech End Users Interview |
Technology End-Users, including patient and advocacy communities |
50 |
1 |
30/60 |
25 |
Totals |
|
|
|
575 |
|
All three groups of respondents comprise primarily of scientists, engineers, and clinicians. The mean hourly wage rates was calculated taking the average of: $42.98 Medical Scientists occupation code 19-1040 (http://www.bls.gov/oes/current/oes_nat.htm#19-0000), $45.18 Biomedical Engineers occupation code 17-2031 (http://www.bls.gov/oes/current/oes172031.htm), and $44.87 Health Diagnosing and Treating Practitioners occupation code 29-1000 (http://www.bls.gov/oes/current/oes_nat.htm#29-0000) from the May 2013 National Occupational Employment and Wage Estimates - United States for the above general categories were averaged and which resulted in $44.34 per hour. The total and annualized cost to respondents is estimated to be $25,495.50 annually (Table A.12-2).
Table A.12-2 Annualized Cost to Respondents
Type of Respondents |
Number of Respondents |
Total Annual Burden Hours |
Hourly Wage Rate* |
Respondent Cost |
IMAT Grantees |
100 |
100 |
$44.34 |
$4,434.00 |
IMAT Grantees and Other NIH Grantees |
900 |
450 |
$44.34 |
$19,953.00 |
Technology End-Users, including patient and advocacy communities |
50 |
25 |
$44.34 |
$1,108.50 |
Total |
$25,495.50 |
|||
There are no direct costs to respondents other than their time to participate in the study.
The total and annualized cost to the Federal Government is approximately $149,570.83 for this information collection. The majority of the expenses are from contractor costs that include non-federal personnel costs including salary and benefits of a project director (contractor), research associates, and a survey statistician for approximately 4 to 5 months. Federal personnel costs include 2% participation from 2 program officers, over 4 to 5 months. Materials costs include hosting charges for Survey Analytics and interview recording supplies. The total costs are in Table A.14-1.
A.14-1 Estimate of Total Cost-Government
Personnel Costs |
Grade/Step |
Annual Rate |
% of time over 12 Months |
Total Cost |
Program Officer |
15/1 |
$126,245 |
2% |
$2,524.90 |
Program Officer |
15/1 |
$126,245 |
2% |
$2,524.90 |
Sub-Total Federal Personnel |
$5,049.80 |
|||
Total Contractor Costs |
$137,521.03 |
|||
Materials Costs |
$7,000.00 |
|||
TOTAL |
$149,570.83 |
|||
This is a new information collection.
Data Analysis
Data quality control and quality assurance procedures will be developed and implemented by senior evaluation professionals and applied to all collected data. This will include procedures to ensure accuracy and consistency in data entry, data manipulation, and calculation. For quantitative data, internal validity will be checked as necessary for analysis. Descriptive and summary statistics will be calculated. If warranted, data from multiple sources may be cross-tabulated to address the study questions. Analytical statistics (e.g., linear regression – especially for publication analysis/bibliometric information) and network statistics (e.g., degree centrality, betweenness, closeness and eigenvector centrality – especially for multi-disciplinary analyses) may be used to assess differences between ICMICs and comparator institutions should comparable information be obtained from the comparators.
For qualitative data, a more sophisticated data management and qualitative analysis software tools (e.g. nVivo software) may be employed (although it is not expected that the volume of qualitative data will require this). Qualitative data will be coded and analyzed using standard qualitative methods.
Plans for Publication
No plans for publication.
Project Time Schedule
The evaluation will be overseen by a trans-NIH evaluation advisory committee (EAC – Attachment 5), and it is estimated that it will take an experienced evaluation team 10-12 months to complete, with the information collection component taking approximately 2-4 months. Provided below is an anticipated timeline of major tasks.
Year 1 (CY 2014)
EAC recruited and accomplish the initial tasks of approving specific elements of the evaluation scope of work (SOW), <2 months (estimate Summer 2014)
Issue SOW by August 2014
Year 2 (CY 2015)
Completion of the following tasks by February 2015:
Identification of all IMAT grant applications (and applicants) and grant awards (and awardees)
Construction of a sample frame of NIH awardees for technology development grants
Completion of the following contractor tasks before April 2015
Development of a database to capture associated output or outcome data related to the IMAT Program
Development of Principal Investigator profiles based on data sources described in §3 of this application
Data analysis of applicants and awardees from the IMAT and related programs
Meeting of EAC, April 2015
Review of archival data analysis
Completion of the following contractor tasks by August 2015
Conduct survey and interview protocols with successful IMAT grant recipients, successive technology owners, NIH awardees (comparison group), and other Federal program staff
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate
No exemption from the display of the OMB Expiration Date are being requested.
There are no exceptions to certification being requested.
1 Though the technology end users are a separate respondent group, it is conceivable that some individuals in that group could also be IMAT awardees.
2 The pilot evaluations involved interviewing 9 individuals for each of the years (2008, 2010, and 2013) and thus did not need OMB clearance.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PLCO SCREENING TRIAL |
| Author | Linda Suit |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |