May 4, 2009
Name: G. Stephane Philogene, Ph.D.
Address: National Institutes of Health
31 Center Drive, Room B1C32
Bethesda, MD 20892-2027
Telephone: 301.402.3902
Fax: 301.402.1150
Email: [email protected]
A.1 Circumstances Making the Collection of Information Necessary 1
A.2 Purpose and Use of the Information 2
A.3 Use of Information Technology and Burden Reduction 2
A.4 Efforts to Identify Duplication and Use of Similar Information 2
A.5 Impact on Small Businesses or Other Small Entities 3
A.6 Consequences of Collecting the Information Less Frequently 3
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 3
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 3
A.9 Explanation of Any Payment or Gift to Respondents 4
A.10 Assurance of Confidentiality Provided to Respondents 4
A.11 Justification for Sensitive Questions 4
A.12 Estimates of Hour Burden Including Annualized Hourly Costs 4
A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers 6
A.14 Annualized Cost to the Federal Government 6
A.15 Explanation for Program Changes or Adjustments 6
A.16 Plans for Tabulation and Publication and Project Time Schedule 6
A.17 Reasons Why Display of OMB Expiration Date is Inappropriate 7
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 7
List of Attachments:
Attachment 1: NIA Program Description and Requirements
Attachment 2a: New Investigator Award Applicant Survey Instrument
Attachment 2b: Extramural Reviewers Informal Discussion Guide
Attachment 3: Privacy Procedures of Contractor
Attachment 4: Respondent Instructions
Attachment 5: Introductory & Follow-up Instructions
This request seeks approval for OMB clearance for the NIH New Innovator Award (NIA) Program Process Evaluation. The data collection consists of an online survey to be completed by participants (applicants and awardees) and a phone survey to be completed by Extramural reviewers. The information gathered from these surveys will document the NIA program operations and will be used as a guide to the program officers in their future strategic and management decisions.
The NIA Program was initiated in 2007 by the Office of the Director (OD) at the National Institutes of Health (NIH) to support exceptionally creative new investigators who propose highly innovative approaches that have the potential to produce an unusually high impact. The program awards are up to $1.5 million (average of $300,000 per year) in direct costs for five years. The objective is to 1) stimulate highly innovative research and support promising new investigators and 2) to allow new investigators who may have exceptionally innovative research ideas, but not the required preliminary data to fare well in the traditional peer review system, to have an opportunity to pursue their ideas. For program description and requirements, see Attachment 1.
NIH wishes to receive OMB clearance to perform an evaluation for the first three years (FY2007, FY2008, and FY2009) of the program. The primary objectives of the study are to: (1) assess the NIA award selection process; (2) determine if the program was implemented as planned; and (3) determine if the process was conducted in accordance with the overall mission of the NIA program.
NIH has tasked the Science and Technology Policy Institute to conduct the process evaluation. The Science and Technology Policy Institute (STPI), one of three federally funded research and development centers run by the Institute for Defense Analyses, assists the Executive Branch of the US government by providing objective, high-quality analytic support to inform program managers and others who run S&T programs.
The unique mode and public nature of the New Innovator Award (NIA) program make its monitoring and evaluation a high priority for the NIH. Additionally, because it is a pilot program, an evaluation is necessary to determine the NIA’s program’s long-term value to the NIH and biomedical science more generally. This submission is for the collection of information for a Process Evaluation (PE) of the NIA program – an evaluation study to determine whether the program is executed according to criteria.
The program is authorized under Sections 301 and 405 of the Public Health Service Act as amended (42 USC 241 and 284) and under Federal Regulations 42 CFR Part 52 and 45 CFR Parts 74 and 92. The proposed data collection supports the management and evaluation of this program.
The purpose of the survey and interview protocol is twofold: to assess the NIA program's selection process and to collect information that could be used by the NIA program officers to guide strategic and management decisions. The survey instruments are in Attachments 2a (Applicant Survey) and 2b (External Reviewer Interview Protocol).
Specifically, information will be collected for the application and review processes:
Application process: activities related to submitting an application package, including: the time provided to complete a package; the form used (Standard Form 424, OMB #0925-0001, expiration date: 11/30/10); and components of the application, including the personal essay.
External review process: activities related to reviewing and scoring applications, including the application of criteria.
Once information from the survey and interviews is gathered and analyzed, a report will be prepared that documents the nature and extent of NIA activities and outputs and overall program efficiency and effectiveness. The findings of the evaluation will be presented to the NIH management and will be used by them to determine how the program and its administration may be altered and improved in future years.
The proposed NIA Applicant Surveys uses Internet technology in order to minimize burden on respondents. All NIA applicants will be sent a link to a Web-based survey that can be completed and submitted on-line. Non-respondents will be followed-up by email and/or telephone. In the process of the survey design significant effort was made to ensure easy access on-line. In addition, the majority of the NIA Applicant Survey questions are closed-ended (e.g. ‘yes/no’) to reduce burden on respondents. Finally, all information that is available from other sources (e.g. participant names and affiliations) will be preloaded, so that respondents will need only to verify it.
The Extramural Reviewers will be interviewed by phone using an interview protocol. To select candidates for these phone interviews, a stratified purposeful sampling strategy will be used. All evaluators are invited to participate, but based on the responses, a convenience sample will be calculated to fill desired strata for analysis; by stratifying the sample, the data collected will be representative. A series of phone interviews are then scheduled by the Science and Technology Policy Institute via email throughout eight weeks to meet the stratified sample targets.
This data collection will be associated with an IT system to collect, use, store, maintain, disclose and possibly transmit data if necessary. Prior to starting data collection, a Privacy Impact Assessment (PIA) will be undertaken with the Privacy Act Coordinator and Information Systems Security Officer to assess privacy and security risks of the IT system.
The proposed survey does not gather data that duplicates other efforts. Because a process evaluation of the NIA program has never been undertaken, information similar to that which we propose to collect is currently unavailable.
The NIA staff has carefully examined the data that are available on individual NIA applicants in order to assure that no information is replicated under the present evaluation. While NIA applicants submitted information on their most significant accomplishments, creativity, motivation, and communication skills, this does not contain answers to questions posed in the survey (some examples are listed below). Factual information from the application process will be preloaded into the survey, as discussed in section A.3.
Only direct contact with program participants can yield information such as:
How participants heard of the NIA program;
If the criteria used to evaluate grants is perceived as useful in identifying “innovators;”
Factors that make a given application successful
Level of satisfaction of participants, both applicants and reviewers.
Where applicants have received previous funding (e.g. have they received grants from other institutions geared toward high-risk research such as Howard Hughes Medical Institute and the Department of Defense’s Defense Advanced Research Projects Agency);
Small businesses are not involved in this study. All survey respondents are biomedical researchers who have participated in the NIA program.
The surveys will only be conducted one time for each group; there will be no periodicity. The NIA program is a relatively new program, and if the information is not collected, NIH will be unable to document the outputs of the NIA program and the NIH will not be able to identify concerns and improve the program.
This project fully complies with all guidelines of 5 CFR 1320.5.
The 60-day Federal Register notice was published (Vol. 74, No. 33, Friday February, 20 2009, pp. 7908-7909). No comments were received.
The Science and Technology Policy Institute had several in-person meetings and conference calls with NIH staff, who provided important background information and made suggestions on the evaluation design. The NIH staff are:
Judith Greenberg, Ph.D. Director, Division of Genetics and Development Biology, National Institute of General Medical Services
Peggy Schnoor, Program Coordinator, NIH Director’s New Innovator Award Program, National Institute of General Medical Services
Shan R. McCollough, Program Coordinator, NIH Director's Pioneer Award Program, National Institute of General Medical Services
Respondents will not be paid for participating in this survey and will not receive any gifts in return for participation. Participation is completely voluntary.
The subjects will be informed that their responses to the questionnaire are to be reviewed by the Science and Technology Policy Institute for purposes of analysis and reporting. Given the nature of the study, assurance of confidentiality is provided to respondents. Some materials may be disseminated in aggregate to the public by NIA staff in order to inform the research community of the results of the study, although no quotes or specific answers will be attributed to individual respondents. NIH’s System of Records and privacy procedures under the Privacy Act are included in the Privacy Act System of Records Notice (Federal Register Vol. 67 No. 187, pages 60741-60794, September 26, 2002).
The NIA Applicant Survey and Extramural Reviewer interview protocol will contain the following general assurance of confidentiality:
“…Your responses will be kept strictly confidential: If you choose to participate, respondent confidentiality will be protected to the extent provided by law, and STPI will report only aggregate information concerning overall impressions of the process to the NIH.”
In order to ensure data security, all employees of the Science and Technology Policy Institute, are required to adhere to strict standards and sign a non-disclosure agreement as a condition of employment (for details, please consult Attachment 3). The Science and Technology Policy Institute has extensive experience collecting information and maintaining its confidentiality and security. All data files on multi-user systems will be under the control of the database manager, with access limited to project staff on a “need to know” basis only. Finally, respondent access to the web-based survey is password-protected.
Data collection instruments will not include any sensitive questions nor any personally identifiable information (PII).
There are no questions of a sensitive nature neither in the survey nor in the interview protocol. In order to understand the background characteristics of applicants, the questionnaire does ask where a given applicant has received previous funding. This information will be used analytically to determine if the NIA program is attracting individuals with “innovator” backgrounds, i.e. has received funding from other programs or institutions that are known to support high-risk/high-impact research. Respondents may not know or may choose not to provide the information that they feel is privileged, such as previous funding sources. As discussed in the previous section, the Science and Technology Policy Institute will hold individual data strictly confidential, and any public reporting of the data will be in aggregate form that will not allow for the identification of individuals.
Every effort will be made to minimize the burden on the respondents. The survey instruments have been reduced to the minimum possible length (e.g., through the use of largely closed-ended question formats). The online survey for Applicants should take, on average, 15 minutes to complete. The phone interview for Extramural Interviewers should take, on average, 30 minutes to complete. For the online applicant survey, each person’s name, address, institution, etc. will be entered on the survey form in advance based on the NIA program records, and space will be available for respondents to make changes where any of this information is incorrect. Respondents (applicants) will be given up to 8 weeks to reply and will only have to fill out one online questionnaire, which will be Web-based. No copies will have to be made or sent. The survey/interviews will be conducted only once, not repeatedly. Respondents do not need to retain or consult records for purposes of this survey.
Further reductions in the instrument length would jeopardize the power of the survey/interview protocol to accurately assess the program. Because program participants come from many institutions and backgrounds, reducing the respondent population would have a negative impact of our ability to identify true differences between important subpopulations: for example, researchers from the social sciences versus natural sciences, or males versus females.
Based on estimated response rates and hour burden from the New Innovator Award pilot surveys, we anticipate the following total burden:
Table 1: Annualized Estimate of Hour Burden |
||||
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time for Response (hr) |
Total Hour Burden* |
Applicants |
570 |
1 |
.25 |
142.5 |
Extramural Reviewers |
92 |
1 |
.50 |
46 |
Total |
662 |
1 |
.28 |
188.5 |
Total Burden = N Respondents*Response Frequency*(minutes to complete/60)
Thus, the expected burden level for this study is 188.5 hours.
An hourly earning rate for participants was estimated using the U.S. Department of Labor, Bureau of Labor Statistics National Compensation Survey (http://www.bls.gov/ncs/ncswage2007.htm).
NIA program officers provided a list of research areas of applicants. Based on this list, relevant professions listed in the NCS (e.g. natural scientists, physicians, and college and university professors or relevant disciplines) were averaged. The average hourly earnings of this group is $46.23. Given that nominators and nominees are likely leading biomedical scientists, we assumed an additional 40% in hourly earnings, for an estimated hourly wage of $64.72. With an average of 15 minutes of time required for applicants to complete the questionnaires and 30 minutes for extramural reviewers, the annual cost for the applicants and reviewers for a New Innovator Award would equal approximately $12,199.72
Table 2: Annualized Cost to Respondents |
||||
Type of Respondents |
Number of Respondents |
Response Frequency |
Approx. Hourly Wage Rate |
Total Respondent Cost** |
Applicants |
570 |
1 |
$64.72 |
$9,226.60 |
Extramural Reviewers |
92 |
1 |
$64.72 |
$2977.12 |
Total |
662 |
1 |
$64.72 |
$12,199.72 |
**Total Respondent Cost = Total Hour Burden * Hourly Wage Rate
There are no capital, maintenance or operating costs to respondents.
A.14.1 Annualized Cost to Contractor
The cost to maintain and implement the survey, including contractor’s fixed fee will be $20,000. This does not include analyses of collected data or preparation of reports.
A.14.1 Estimate of Other Total Annual Cost Burden to Contractor |
|
Survey maintenance and software licenses |
$5,000 |
Website design and support |
$5,000 |
Record keeping and follow up |
$10,000 |
Total |
$20,000 |
A.14.2 Annualized Cost to the Federal Government
Annualized Cost to the Federal Government is composed, in part, of an aggregate estimate from Items A.12 and the information above, as this is a one-time survey that will require less than one year to complete. In addition, there are costs of the NIA Project Officer, NIH OMB Clearance Officer, other NIA professional staff, and support staff time. Based upon a discussion with the Project Officer, we have estimated that approximately a quarter of a year’s time is required in association with the conduct of $52,199.72 (Table A.14.2).
A.14.2 Total Cost Burden of Information Collection |
|
Annualized Cost to Respondents |
$12,199.72 |
Other Annual Cost to Contractor (from A.14.1) |
$20,000 |
NIH/NIA Staff Time |
$20,000 |
Total |
$52,199.72 |
The NIA Process Evaluation is a new request. Based on the results of previous annual collections and ongoing trends of evaluation of the NIH Director’s Pioneer Award (NDPA), we estimate that the number of survey respondents will be in the range of 470 to 670 with an estimated average of about 570 applicants. (The OMB Clearance number for the NPDA process evaluation is OMB No. 0925-0534.)
The NIA Process Evaluation began in August 2008, and will end in August 2010. The NIA Project Officer has reviewed and approved the survey instrument prior to its implementation. The survey will be conducted during the summer of 2009 for years 1 and 2 of the NIA program. The survey will be conducted in the Fall of 2009 for year 3 of the NIA program. The evaluation contractors are required to deliver a draft final report on the evaluation by August 2010.
A.16 Estimated Annual Project Time Schedule |
|
Activity |
Time Schedule |
FY07 Survey implementation & Interviews conducted |
1-2 months after OMB approval (July-August, 2009) |
FY07 Data analyses |
4-5 months after OMB approval (September-October 2009) |
FY07 Report writing, dissemination |
8-9 months after OMB approval (November-December 2009) |
FY08 Survey implementation & Interviews conducted |
1-2 months after OMB approval (July-August, 2009) |
FY08 Data analyses |
4-5 months after OMB approval (September-October 2010) |
FY08 Report writing, dissemination |
8-9 months after OMB approval (November-December 2010) |
FY09 Survey implementation & Interviews conducted |
6-7 months after OMB approval (January-February, 2010) |
FY09 Data analyses |
6-7 months after OMB approval (March-April 2010) |
FY09 Report writing, dissemination |
8-9 months after OMB approval (May-June 2010) |
No exceptions are sought; the OMB Expiration Date will be displayed on the survey instruments.
No exceptions are sought from the Paperwork Reduction Act.
| File Type | application/msword |
| File Title | Supporting Statement for the |
| Author | sshipp |
| Last Modified By | Guest |
| File Modified | 2009-05-04 |
| File Created | 2009-05-04 |