September 26, 2012
Ravi Basavappa, Ph.D.
Office of Strategic Coordination
Division of Program Coordination, Planning, and Strategic Initiatives
Office of the Director, NIH
1 Center Drive, MSC 0189
Building 1, Room 203
Bethesda, MD 20892-0189
Telephone: 301-594-8190
Fax: 301-435-7268
Email: [email protected]
A.1 Circumstances Making the Collection of Information Necessary 1
A.2 Purpose and Use of the Information 2
A.3 Use of Information Technology and Burden Reduction 3
A.4 Efforts to Identify Duplication and Use of Similar Information 3
A.5 Impact on Small Businesses or Other Small Entities 3
A.6 Consequences of Collecting the Information Less Frequently 3
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 4
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 4
A.9 Explanation of Any Payment or Gift to Respondents 4
A.10 Assurance of Confidentiality Provided to Respondents 4
A.11 Justification for Sensitive Questions 5
A.12 Estimates of Hour Burden Including Annualized Hourly Costs 6
A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers 7
A.14 Annualized Cost to the Federal Government 8
A.15 Explanation for Program Changes or Adjustments 8
A.16 Plans for Tabulation and Publication and Project Time Schedule 8
A.17 Reasons Why Display of OMB Expiration Date is Inappropriate 9
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 9
List of Attachments
Attachment A.1: 2012 Request for Applications (RFA)
Attachment A.2: Paper Survey for Editorial Board of Reviewers
Attachment A.3.1: Customer Satisfaction Online Survey for Applicants: Principal Investigators (Versions A & B)
Attachment A.3.2: Customer Satisfaction Online Survey for Applicants: Officials of Host Institutions
Attachment A.3.3: Screenshots Customer Satisfaction Online Survey for Applicants (Versions A & B)
Attachment A.3.4: Screenshots Customer Satisfaction Online Survey for Host Institutions
Attachment A.4: Paper Survey for Early Independence Principal Investigator
Attachment A.5: Phone Interview Questions for Early Independence Principal Investigator (Versions A & B)
Attachment A.6: Online Survey for Early Independence Principal Investigator
Attachment A.6.1: Screenshots Online Survey for Early Independence Principal Investigator
Attachment A.7: Phone Interview Questions for Point of Contact at Host Institution
Attachment A.8: Online Survey for Point of Contact at Host Institution
Attachment A.8.1: Screenshots Online Survey for Point of Contact at Host Institution
Attachment A.9: Members of the NIH Common Fund High-Risk High-Reward Working Group for the EIA Program
Attachment A.10: Letter from the NIH Office of Human Subjects Research (OHSR)
Attachment A.11: Letter from the NIH Privacy Officer
This request seeks approval for OMB clearance to conduct a Process Evaluation of the NIH Director’s Early Independence Award (EIA) Program. This request for clearance includes data collection efforts for applicants, extramural reviewers, and awardees. The data collection for applicants consists of two online customer satisfaction surveys. The purpose of the online customer satisfaction surveys is to gather opinions and give applicants an opportunity to share their experiences. The information from these surveys may be used to make changes to subsequent Requests for Applications (RFA). The data collection for reviewers consists of a paper survey with questions about the EIA selection process. The data collection for awardees consists of a paper survey, phone interviews, and online surveys. The purpose of the data collection for awardees is to assess how the program is being implemented, assess progress being made by the EIA Principal Investigators (PIs), and assess support being provided by the Host Institution. The information gathered from applicants, reviewers, and awardees will document the EIA program operations and activities, and will be used to improve the program and guide future strategic and management decisions.
The surveys will help fulfill the requirements of:
• Executive Order 12862, “Setting Customer Service Standards,” which directs Agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector; and
• The March 3, 1998 White House Memorandum, “Conducting Conversations with America to Further Improve Customer Service,” which directs Agencies to determine the kind and quality of service its customers want as well as their level of satisfaction with existing services.
The NIH Early Independence Award (EIA), a NIH Common Fund program, was established in FY 2011. The NIH Common Fund is managed by the Office of Strategic Coordination (OSC), part of the Division of Program Coordination, Planning and Strategic Initiatives (DPCPSI), located in the Office of the Director (OD). The EIA program provides an opportunity for exceptional junior scientists, who have already established a record of innovation and research productivity, to launch an independent research program and skip post-doctoral training. It is also an opportunity for research institutions to invigorate their research departments by recruiting and hosting outstanding, early career scientists.
The goal of the EIA Program is to accelerate the entry of exceptional junior investigators into positions of independent research. The program distributes $4,000,000 in new awards each year. The program was established because recent trends have demonstrated an increase in the length of the traditional scientific training period with a concomitant increase in the time it takes for scientists to establish independent research careers. The EIA is an experimental program expected to have at least five annual competitions. The EIA uses a unique selection process designed to select junior scientists who are exceptional investigators and wish to pursue independent research directly after completion of their terminal doctoral/research degree or clinical residency, thereby foregoing the traditional post-doctoral training period. Since its launch in FY 2011, the award has been given to ten junior scientists. For a detailed description of the program and its requirements, please see the 2012 Request for Applications (RFA-RM-11-007) located in Attachment A.1.
NIH wishes to receive OMB clearance to perform a process evaluation for the first four years (FY2011 – FY2014) of the program. The primary objectives of the study are to: (1) assess if the RFAs are meeting the needs of applicants; (2) document the selection process; (3) document EIA program operations and activities; (4) assess the progress being made by the Early Independence Principal Investigators; and (5) assess the support provided by the Host Institutions to the Early Independence Principal Investigators. The high profile and public nature of the EIA program make its evaluation a high priority for the NIH. An evaluation is necessary to determine the EIA program’s long-term value to the NIH and more generally to biomedical science.
Data will be collected from the editorial board reviewers and applicants before the NIH award is granted. The purpose of the paper survey for the editorial board reviewers is to gather comments and document activities related to the selection process (See Attachment A.2). The purpose of the online customer satisfaction surveys for applicants is to gather opinions about the program, and give applicants an opportunity to share their experiences and provide feedback. Since the EIA grant application must be submitted by the Institution hosting the Principal Investigator (i.e., junior scientist), two online surveys have been developed for applicants: one for the Principal Investigator (hereafter referred to as “customer satisfaction survey for applicants”-see Attachment A.3.1) and another one for the Official of the Host Institution (hereafter referred to as “customer satisfaction survey for Host Institutions” see Attachment A.3.2). The information collected may be used to make adjustments to subsequent RFAs. The screenshots of the online customer satisfaction survey for applicants is in Attachment A.3.3 and the screenshots of the online customer satisfaction survey for Host Institutions is in Attachment A.3.4.
Data will be collected from Early Independence Principal Investigators and the Points of Contact at the Host Institutions after they receive the EIA grant. The purpose of the data collection for awardees is to document program operations and activities, assess the progress being made by the Early Independence Principal Investigators, and assess the support provided by the Host Institutions to the Early Independence Principal Investigators.
The methods to gather data from Early Independence Principal Investigators are described as follows. A paper survey will be administered at the beginning of the first year of award (see Attachment A.4). Phone interviews will be conducted at the end of the first year of award (see Attachment A.5). Finally, an online survey will be administered at the end of the second and third year of award (see Attachment A.6). The following methods will be used to gather data from the Point of Contact at the Host Institution. A phone interview will be conducted at the end of the first year of award (see Attachment A.7), and an online survey will be administered the second and third year of award (see Attachment A.8).
Information collected during the EIA process evaluation will be used in multiple ways. First, the information will be used to guide the decisions to modify and improve the program. Second, the results of the evaluation will inform the NIH Director and EIA program staff on the progress of the EIA program. Third, several of the NIH Institutes and Centers can benefit from the lessons learned from the EIA process evaluation about program implementation and design. Fourth, due to its high visibility, the research community has shown substantial interest in the EIA program and the results of the process evaluation will be distributed to these wider audiences. To disseminate the evaluation findings to the evaluation and the scientific communities, efforts also will be made to publish the results of the process evaluation in a professional journal and to present the findings at conferences.
The surveys for applicants and awardees are online to reduce burden. The surveys for applicants include: customer satisfaction survey for applicants (Principal Investigators), customer satisfaction survey for Officials of Host Institutions. The surveys for awardees include: survey for the Early Independence Principal Investigator, and survey for the Point of Contact at his/her Host Institution.
All participants will receive a link to the online survey and a user code to access it. As appropriate, the online surveys will use skip-patterns so that each respondent is only presented with questions relevant to his or her specific situation. In addition, several questions on the surveys are multiple choice or closed-ended to reduce burden on respondents.
The paper survey for the Early Independence Principal Investigators has been developed, to the extent possible, to contain questions that are multiple choice and closed-ended. The questions for both the phone interview with the Early Independence Principal Investigator and the phone interview with the Point of Contact at his/her Host Institution have been designed to reduce respondent burden. The phone interviews include multiple choice and close-ended questions.
Data collection will be associated with an IT system to collect, use, store, and maintain data. A Privacy Impact Assessment (PIA) is currently being done for the online surveys. The NIH Privacy Act Officer and the NIH Information Systems Security Officer are assessing privacy and security risks of the IT system.
There is no similar information available. The data to be collected from applicants, reviewers, and awardees do not duplicate other data collection efforts. The NIH Director's Early Independence Award (EIA) is a new program, established by Dr. Francis Collins, the NIH Director, to combat a recent trend of long-term training periods that increase the time it takes a scientist to establish an independent research career. The EIA program provides a mechanism for exceptional, early career scientists to omit traditional post-doctoral training, and move into independent research positions at U.S. institutions directly upon completion of their graduate degrees (Ph.D, M.D. or equivalent).
Small businesses are not involved in this study.
If the customer satisfaction surveys for applicants are not conducted, NIH will not be able to identify the gaps and barriers the applicants face when submitting the grant application. Also, if the surveys are not conducted, the EIA program will not be adapted to meet the applicant’s needs and satisfaction with existing components will not be ascertained.
If information is not collected from extramural reviewers, NIH will not be able to get feedback about the selection review process, and will not be able to take the steps to improve the process.
If the data from awardees (Early Independence Principal Investigators and the Point of Contact at their Host Institution) are not collected, NIH will not be able to make improvements to the program. Also, the NIH will not be able to assess how the program is being implemented, determine the progress being made by the Early Independence Principal Investigators, and how Host Institutions are contributing to the success of the awardees. Collection on a less frequent basis would reduce the practical utility of the information and inhibit the ability to monitor changes.
This project fully complies with all guidelines of 5 CFR 1320.5.
A.8.1. Comments in Response to the 60-day Federal Register
The 60-day Federal Register Notice was published on June 13, 2012 (Vol. 77, No 114). No comments were received.
A.8.2. Comments in Response to Consultations Outside Agency
Dr. Madeleine F. Wallace, from Windrose Vision, was consulted on the design of the study. Dr. Wallace was contracted by the Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI). The plan for the process evaluation was developed in consultation with NIH Common Fund High-Risk High-Reward Working Group for the EIA program. The list of members is in Attachment A.9. The proposed design is grounded in extensive background research and discussions with NIH program staff and evaluators with similar programs.
Applicants, reviewers, and awardees will not be paid for participating in the data collection efforts and will not receive any gifts in return for participation. Participation is completely voluntary.
The NIH Office of Human Subjects Research (OHSR) has reviewed the process evaluation of the EIA program and determined that the project is not research. The OHSR determination of Not Human Subjects Research is based on the interpretation of 45 CFR 46 and Guidance of Engagement of Institutions in Human Subjects Research. (See letter from the NIH Office of Human Subject Research in Attachment A.10).
The NIH Privacy Act Officer has reviewed this OMB request and determined that the data collection is covered by NIH Privacy Act Systems of Record Notice 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD.” (See letter from the NIH Privacy Officer in Attachment A.11).
The applicants, reviewers, and awardees will be informed that their responses to the data collection efforts are only reviewed by the contractor implementing the evaluation for purposes of analysis and reporting. Given the nature of the study, assurance of privacy will be provided to respondents. Some materials may be disseminated in aggregate to the public in order to inform the research community of the results of the study.
The contractor conducting the study will be required to adhere to the following safeguarding procedures:
The safeguarding protections offered to all participants will be included in the invitations and introduction to the survey instruments. Respondents will be informed that their participation is voluntary and that no consequences will be associated with either responding or not responding. For example, the online survey for the Early Independence Principal Investigator and the online survey for the Point of Contact at the Host Institution will have the following statement: “Your participation is voluntary and non-participation will have no impact on you or your institution. If you choose to participate, your responses will be kept private under the Privacy Act, and will not be disclosed to anyone but the researchers conducting this study, except as otherwise required by law. You will not be identified by name and information from the study will only be reported in the aggregate. Your responses will be combined with those of other respondents in the final report.”
Access to data will be restricted to staff working on the project. All paper files (such as paper surveys and handwritten interview notes) shall be stored in locked cabinets.
Names and other identifiable information shall be redacted in all primary data (surveys and interview notes) and replaced with identifier numbers.
All data shall be reported in aggregate and will not contain any identifying information.
The contractor will follow the NIH Privacy Act Systems of Record Notice 09-25-0156.
All computer-based systems will comply with the Privacy Act.
The data collection instruments for reviewers and the customer satisfaction surveys for applicants do not contain any sensitive questions. Reviewers and applicants will not be asked to provide their names, affiliation, gender, or ethnicity.
Given the diversity of its awardees, it is important for NIH to collect data from a wide range of awardees. The two phone interviews (one with the Early Independence Principal Investigator and the other one with the Point of Contact at the Host Institution) will include the name and affiliation. This information will be removed for analysis and reported in aggregate form. Only the online survey for the Early Independence Principal Investigators includes demographic questions such as race, ethnicity, and gender. This information will allow NIH to analyze the survey data by subgroups and support NIH’s long-standing efforts to strengthen the diversity of its applicants. Given that the EIA is a new program, it is important to gather demographics to inform outreach efforts.
Early Independence Principal Investigators may skip any or all of the Personally Identifiable Information (PII) questions concerning race, ethnicity, and gender. Those who choose to provide these demographic data will do so on a strictly voluntary basis. The data will be reported in aggregate form that will not allow for the identification of individuals.
As requested by the NIH Privacy Act Officer, the Privacy Act Systems of Record Notice 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” will be distributed to staff responsible for handling any PII. The contractor that will conduct the evaluation and the NIH Center for Information Technology (CIT) that will develop the online surveys will comply with the Privacy Act. (See letter from NIH Privacy Officer in Attachment A.11).
A.12.1 Estimates of Hours Burden
Every effort will be made to minimize the burden on the respondents.
Applicants - The two online customer satisfaction surveys for applicants: one for Principal Investigators and the other one for Officials of Host Institutions should take on average 15 minutes each to complete. On April 4, 2012, these two surveys were approved to gather data in April and May of 2012 under OMB # 0925-0476. If this current request for clearance is approved, the same surveys will be used to gather data from applicants for the duration of the clearance (2013, 2014, and 2015).
Reviewers -The paper survey for editorial board reviewers will be administered when the review process is completed. The survey should take approximately 15 minutes to complete.
Awardees -The paper survey for the Early Independence Principal Investigators should take to complete on average 30 minutes. The paper survey will be administered at the beginning of the first year of the award when the Early Independence Principal Investigators are onsite for their first awardee meeting at NIH.
The phone interview with the Early Independence Principal Investigator and the phone interview with the Point of Contact at the Host Institution should take on average 60 minutes each to complete. Both phone interviews will be conducted within a two-month timeframe and will be conducted only once.
The online surveys for the Early Independence Principal Investigators and the Points of Contact at Host Institutions should take on average 60 minutes to complete. The online surveys will be open for a one-month period, and they will be conducted at the end of the second and third year of the award.
Further reductions in the paper survey, the phone interviews, and the online surveys would jeopardize accurate assessment of the program. Table A.12.1 displays the annualized estimate of hour burden. The expected burden level for this study is 158 hours.
A.12.1: Annualized Estimate of Hour Burden |
||||
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Response (in hrs.) |
Annual Hour Burden |
Editorial Board Reviewers (paper survey) |
15 |
1 |
15/60 |
4 |
Applicants – Principal Investigators (online survey) |
150 |
1 |
15/60 |
38 |
Applicants – Officials of Host Institutions (online survey) |
150 |
1 |
15/60 |
38 |
Awardees – Early Independence Principal Investigator (paper survey – beginning of 1st yr. of award) |
12 |
1 |
30/60 |
6 |
Awardees – Early Independence Principal Investigator (phone interview – end of 1st yr. of award) |
12 |
1 |
1 |
12 |
Awardees – Early Independence Principal Investigator (online survey – end of 2nd, and 3rd yr. of award) |
24 |
1 |
1 |
24 |
Awardees – Point of Contact at Host Institution (phone interview – end of 1st yr. of award) |
12 |
1 |
1 |
12 |
Awardees – Point of Contact at Host Institution (online survey – end of 2nd, and 3rd yr. of award) |
24 |
1 |
1 |
24 |
Total |
|
158 |
||
A.12.2 Annualized Cost to Respondents
An hourly earning rate for participants was estimated using the U.S. Department of Labor, Bureau of Labor Statistics National Compensation Survey (NCS) (http://www.bls.gov/ncs/ncswage2010.htm).
NIH staff for the EIA program provided a list of research areas of applicants. Based on this list, relevant professions listed in the NCS (e.g. natural scientists, physicians, and college and university professors of relevant disciplines) were averaged. The average hourly earnings for Principal Investigators is $38.13. Given the variation in geographical locations for the Early Independence Principal Investigators, we assumed an additional 20% in hourly earnings, for an estimated hourly wage of $45.76. For the Institution Officials and Reviewers, we assumed average hourly earnings of $78.37 based on the relevant professions listed in the NCS. The annual cost for the reviewers, applicants, and awardees to participate in the EIA Process Evaluation would equal approximately $9,774.
A.12.2: Annualized Cost to Respondents |
|||
Type of Respondents |
Annual Hour Burden |
Approx. Hourly Wage Rate |
Total Cost |
Editorial Board Reviewers (paper survey) |
4 |
$78.37 |
$313 |
Applicants – Principal Investigators (online survey) |
38 |
$45.76 |
$1,739 |
Applicants – Officials of Host Institutions (online survey) |
38 |
$78.37 |
$2,978 |
Awardees – Early Independence Principal Investigator (paper survey – beginning of 1st yr. of award) |
6 |
$45.76 |
$275 |
Awardees – Early Independence Principal Investigator (phone interview – end of 1st yr. of award) |
12
|
$45.76 |
$549 |
Awardees – Early Independence Principal Investigator (online survey – end of 2nd, and 3rd yr. of award) |
24 |
$45.76 |
$1,098 |
Awardees – Point of Contact at Host Institution (phone interview – end of 1st yr. of award) |
12 |
$78.37 |
$940 |
Awardees – Point of Contact at Host Institution (online survey – end of 2nd, and 3rd yr. of award) |
24 |
$78.37 |
$1,881 |
Total |
158 |
|
$9,774 |
There are no capital, maintenance or operating costs to respondents.
The annual cost to the Federal Government includes both the annual contractor cost and the cost of federal personnel time. The annual total cost to the government is $ 55,313 ($29,560 annual contractor cost and $25,753 annual NIH staff time cost).
Assume the following annual contract costs. The cost for each of five online surveys is estimated to vary between $1,000–$5,000, depending on the size, maintenance, and implementation. The total is estimated to be $25,000 (5 X $5,000). The cost of the administration of the paper survey is estimated to be $340 (1 hour for a senior contractor at $190, and 1 hour for a contractor at $150). The cost to conduct the phone interviews is estimated to be $4,560 (approximately 24 interviews at a rate of $190). Using these estimates, the total approximate cost is $29,560. The estimates provided do not include the preparation of the tools, scheduling, analyses of collected data, or preparation of reports.
Total federal government personnel costs will be $25,753. It is anticipated that three NIH staff from the Office of Strategic Coordination will be involved at the GS-14, GS-13, and GS-12 levels. The costs assumes one GS-14 with a median annual salary of $119,238 working at a level of effort of 5% ($ 5,961); one GS-13 with a median salary of $100,904 working at a level of effort of 7% ($7,063); and one GS-12 with a median salary of $84,855 working at a level of effort of 15% ($12,728). Salaries are based on the January 2012 General Schedule for the Washington, DC Metropolitan area (http://www.opm.gov/oca/12tables/pdf/DCB.pdf) and no estimated increases are expected in 2013.
This is a new collection of information.
The data collection and analysis will be conducted by outside contractors that will work with the NIH Project Officer. To provide the NIH understanding of the implementation of the EIA program, the contractor will prepare a report for the NIH that describes the study and findings.
A.16 Estimated Annual Project Time Schedule |
|
Activity |
Time Schedule |
Administer paper survey for new FY2012 Early Independence Principal Investigators |
1-2 months after OMB approval |
Set up and conduct phone interview with FY2011 Early Independence Principal Investigator and interview with Point of Contact at Host Institution |
1-2 months after OMB approval |
Administer customer satisfaction surveys to FY2013 applicants and officials of Host Institutions |
6-8 months after OMB approval |
Administer paper survey for editorial board reviewers in FY2013 |
8-9 months after OMB approval |
Administer paper survey for new FY2013 Early Independence Principal Investigators |
13 months after OMB approval |
Set up and conduct phone interview with FY2012 Early Independence Principal Investigator and interview with Point of Contact at Host Institution (1st year) |
12-13 months after OMB approval |
Administer online survey to FY2011 Early Independence Principal Investigators and Points of Contact at Host Institutions |
12-13 months after OMB approval |
Analyze data and Report results |
13-18 months after OMB approval |
Administer customer satisfaction surveys to FY2014 applicants and officials of Host Institutions |
18-19 months after OMB approval |
Administer paper survey for new FY2014 Early Independence Principal Investigators |
24-25 months after OMB approval |
Set up and conduct phone interview with FY2013 Early Independence Principal Investigator and interview with Point of Contact at Host Institution (1st year) |
24-25 months after OMB approval |
Administer online survey of FY2011(3rd year) and FY2012 (2nd year) Early Independence Principal Investigators and Points of Contact at Host Institutions |
24-25 months after OMB approval |
Analyze data and Report results |
24-28 months after OMB approval |
Administer customer satisfaction surveys to FY2015 applicants and officials of Host Institutions |
30-32 months after OMB approval |
Administer paper survey for new FY2015 Early Independence Principal Investigators |
35-36 months after OMB approval |
Set up and conduct phone interview with FY2014 Early Independence Principal Investigator and interview with Point of Contact at Host Institution |
35-36 months after OMB approval |
Administer online survey of FY2012(3rd year) and FY2013 (2st year) Early Independence Principal Investigators and Points of Contact at Host Institutions |
35-36 months after OMB approval |
Analyze data and Report results |
36 months after OMB approval |
No exceptions are sought; data collection instruments will display the OMB Expiration Date.
No exceptions are sought from the Paperwork Reduction Act.
| File Type | application/msword |
| File Title | Supporting Statement for the |
| Author | sshipp |
| Last Modified By | Perryman |
| File Modified | 2012-09-26 |
| File Created | 2012-09-26 |