Supporting Statement B for
A Process Evaluation of the NIH Director’s Early Independence Award Program for the Office of Strategic Coordination, an office of the Division of Program Coordination, Planning, and Strategic Initiatives, Office of the NIH Director
August 29, 2012
Ravi Basavappa, Ph.D.
Office of Strategic Coordination
Division of Program Coordination, Planning, and Strategic Initiatives
Office of the Director, NIH
1 Center Drive, MSC 0189
Building 1, Room 203
Bethesda, MD 20892-0189
Telephone: 301-594-8190
Fax: 301-435-7268
Email: [email protected]
B. Collection of Information Employing Statistical Methods 1
B.1 Respondent Universe and Sampling Methods 1
B. 2. Procedures for the Collection of Information 1
B.3. Methods to Maximize Response Rates 3
List of Attachments:
Attachment B.1.1: Protocols for Customer Satisfaction Online Survey for Applicants: Principal Investigators
Attachment B.1.2: Protocols for Customer Satisfaction Online Survey for Applicants: Officials of Host Institutions
Attachment B.2.1: Protocols for Online Survey for Early Independence Principal Investigators
Attachment B.2.2: Protocols for Online Surveys for Point of Contact at Host Institution
Attachment B.3: Protocol for Paper Survey for Early Independence Principal Investigators
Attachment B.4.1: Protocol for Phone Interview with Early Independence Principal Investigator
Attachment B.4.2: Protocol for Phone Interview with Point of Contact at Host Institution
Supporting Statement for the
Paperwork Reduction Act Submission
B: Justification
National Institutes of Health
A Process Evaluation of the NIH Director’s Early Independence Award (EIA) Program
This request seeks approval for OMB clearance to conduct a Process Evaluation of the NIH Director’s Early Independence Award (EIA) Program. This request for clearance includes data collection efforts for applicants, extramural reviewers, and awardees. The data collection for applicants consists of two online customer satisfaction surveys. The purpose of the online customer satisfaction surveys is to gather opinions and give applicants an opportunity to share their experiences. The information from these surveys may be used to make changes to subsequent Requests for Applications (RFA). The data collection for reviewers consists of a paper survey with questions about the EIA selection process. The data collection for awardees consists of a paper survey, phone interviews, and online surveys. The purpose of the data collection for awardees is to assess how the program is being implemented, assess progress being made by the EIA Principal Investigators (PIs), and assess support being provided by the Host Institution. The information gathered from applicants, reviewers, and awardees will document the EIA program operations and activities, and will be used to improve the program and guide future strategic and management decisions.
The universe of respondents for which the clearance is sought includes: reviewers; applicants, and awardees. A census will be used for each data collection effort. Conducting a census is appropriate because sampling would result in the number of respondents from important sub-groups being too small to permit comparative analyses.
Types of data collection for applicants - The data collection for applicants consists of two online surveys. The two customer satisfaction surveys for applicants will be administered after applications have been submitted to NIH.
Types of data collection for awardees - The data collection for awardees consists of five types of data collection: (1) A paper survey will be administered to EIA Principal Investigators after they receive the award. (2) Phone interviews with EIA Principal Investigators will be conducted at the end of the first year of award. (3) An online survey for EIA Principal Investigators will be administered at the end of the second and third year of the award. (4) Phone interviews with the Point of Contact at the Host Institution will be conducted at the end of the first year of the award. (5) An online survey for the Point of Contact at the Host Institution will be administered at the end of the second and third year of the award. Before data collection begins, NIH EIA Program Director will contact all EIA awardees (Principal Investigators and the Points of Contact at Host Institutions) to describe the purpose of the process evaluation and introduce the contractor for the evaluation.
Procedures for online surveys for applicants and awardees - The surveys for applicants include: one for the Principal Investigator (hereafter referred to as “customer satisfaction survey for applicants”); and one for the Official of the Host Institution (hereafter referred to as “customer satisfaction survey for Host Institutions). Attachment B.1.1 contains protocols for the customer satisfaction survey for applicants; Attachment B.1.2 contains protocols for the customer satisfaction survey for Host Institutions.
The online surveys for awardees include: survey for the EIA Principal Investigator (hereafter referred to as “Survey for Early Independence Principal Investigators”), and survey for the Point of Contact at his/her Host Institution (hereafter referred to as “Survey for Point of Contact at EIA Host Institution”). Attachment B.2.1 contains protocols for the online survey for Early Independence Principal Investigators. Attachment B.2.2 contains protocols for the online survey for Point of Contact at EIA Host Institution.
The following procedures will be used for all online surveys.
The online surveys will be designed to be clear and easy to navigate. As appropriate, the online surveys will use a skip-pattern so that each respondent is only presented with questions that are relevant to his or her specific situation.
An e-mail invitation will be sent to participants. The email will explain the purpose of the online survey and provide a hyperlink to the survey website. Immediately afterwards, in a separate email, a user code will be sent to all participants for them to access the survey online.
One week after the e-mail invitation, a reminder e-mail will be sent to all non-respondents. The e-mail will encourage those who have not yet logged into the website to participate in the survey. Immediately afterwards, a separate email containing a user code will be sent to non-respondents for them to access the survey online.
One week after the first reminder e-mail, a second e-mail reminder will be sent to all non-respondents. The e-mail will reinforce the purpose and relevance of the survey. Immediately afterwards, a separate email containing a user code will be sent to the non-respondents for them to access the survey online.
Procedures for paper survey for editorial board reviewers - The paper survey for the editorial board reviewers will be administered after the face-to-face review meeting is completed. The survey will be included in the review meeting package. The NIH Scientific Review Officer will inform the editorial board reviewers that their packages include a survey to gather opinions about the EIA review process. The meeting's agenda will include a time for editorial board reviewers to complete the survey. An independent contractor will explain the purpose of the survey, instruct them not to include their names on the survey, collect the completed surveys, and be available to answer any questions.
Procedures for paper survey for awardees - The paper survey for the Early Independence Principal Investigators will be administered when the awardees are onsite at NIH campus during the introductory and welcome meeting. Respondents will be informed of the purpose of the survey and that their participation is voluntary. Attachment B.3 contains the protocol.
Procedures for Phone Interviews with awardees. Interviews with Early Independence Principal Investigators will be conducted by telephone using an interview protocol. The interviews will be conducted within a two-month period. Interviewees will be contacted in advance to ensure they have sufficient notice to schedule the phone interview. Basic information about the interviewee will be collected in advance to the extent possible (e.g., title, affiliation, research topic) and verified with respondents during the interview. The same procedures will be used for the phone interview with the Point of Contact at the Host Institution. Attachment B.4.1 contains the protocol for Early Independence Principal Investigators; and Attachment B.4.2 contains the protocols for the Point of Contact at Host Institutions.
Content analysis will be used to analyze the qualitative data collected from surveys and phone interviews. As appropriate, descriptive statistics such as counts, ranges, means, and frequency distributions will be used to report on multiple choice questions and scales. Once the analyses are completed, a thorough documentation of program activities, findings, and conclusions will be included in the final report. Specific recommendations will be also reported.
The process evaluation of the EIA program will make an important contribution to the program evaluation field as it will explore data collection and analysis strategies and the approaches that can be adapted to evaluate other programs at the NIH. To disseminate the evaluation findings to the evaluation and the scientific communities, efforts also will be made to publish the results of the process evaluation in a professional journal and to present results at conferences.
Online surveys. Consistent with sound survey methodology, the design of all online surveys will include approaches to maximize response rates, while retaining the voluntary nature of the effort. We will use the following approaches for all online surveys:
The online surveys will be designed to be clear and easy to navigate. As appropriate, the online surveys will use a skip-pattern so that each respondent is only presented with questions that are relevant to his or her specific situation. Also, the online survey contains multiple choice and close-ended questions.
The introductory e-mail invitations for applicants (one for the Principal Investigator and one for the Official of the Host Institution) will inform participants that the purpose is to gather opinions about the EIA program. The email for applicants will contain enough information to generate interest in the online surveys. The emails will provide a point of contact for additional information. In 2011 similar version of the customer satisfaction surveys were used and obtained a response rate of 79% (OMB# 0925-0476).
Advanced outreach to awardees will raise awareness about the online surveys and encourage participation. The purpose of the process evaluation will be explained at the welcome meeting for EIA awardees at the NIH campus. At the meeting in December, 2011 for the FY 2011 awardees, the NIH Director, Dr. Francis Collins, stated that this program will be evaluated. The e-mail invitation for awardees (one for EIA Principal Investigator and one for the Point of Contact at the Host Institution) will also inform participants of the purpose of the process evaluation. The emails will provide a point of contact for additional information.
Follow-up e-mails will remind participants about the online surveys and encourage participation. These reminders will always include a link to the survey.
As an alternative mode for answering the questions, a final reminder letter and a hardcopy version of the online survey will be sent to awardees who have not responded.
Paper survey for the Editorial Board Reviewers. We will use the following approaches:
The survey has been designed to take an average of 15 minutes to complete.
The survey has been designed to be clear and easy to understand.
The survey will be included in the package for the review meeting. The NIH Scientific Review Officer will inform the Editorial Board Reviewers that their packages include a survey to gather opinions about the EIA selection process.
The meeting's agenda will include a time for Editorial Board Reviewers to complete the survey. An independent contractor will explain the purpose of the survey, instruct them not to include their names on the survey, collect the completed surveys, and be available to answer any questions.
Editorial Board Reviewers who are not able to complete the survey during the allocated time will be given a self-addressed stamped envelope and asked to return the survey within 2 days.
Phone Interviews. To maximize the response rates for the phone interviews with awardees (EIA Principal Investigators and Point of Contact at Host Institutions) sufficient time for data collection will be provided. Phone interviews will be carried out over the course of two months to make sure that the busy schedules of respondents can be accommodated. Also, emails reminders will be sent to respondents to encourage participation in the phone interviews.
Online surveys for applicants. The two surveys were approved under OMB # 0925-0476 on April 4, 2012 to gather data in April and May of 2012. If this current request for clearance is approved, the same surveys will be used to gather data from applicants for the duration of the clearance (2013, 2014, and 2015).
Reviewers. Based on 9 responses, revisions were made to the survey. The NIH staff also provided technical feedback.
Awardees. Based on 9 responses, revisions were made to the paper survey for Early Independence Principal Investigators. The questions for the phone interviews and the online surveys have been pilot tested with fewer than 9 Early Independence Principal Investigators and Points of Contact at Host Institutions. They provided feedback on the content and commented on the clarity of the draft questions. Their participation in the pre-testing was voluntary and anonymous. Feedback was collected via email.
The NIH EIA program staff and the NIH Common Fund High-Risk High-Reward Working Group for the EIA program provided technical feedback to the data collection instruments for the applicants, reviewers, and awardees.
An independent contractor participated in the development of the design for this process evaluation study. This contractor has extensive experience in statistical methods; evaluation of NIH programs; and development of surveys for NIH applicants, reviewers, and awardees. Her name is Madeleine F. Wallace, Ph.D., and she can be contacted either by email at [email protected] or phone at (202) 904-5220.
| File Type | application/msword |
| File Title | NDPA Proc Eval 30 day Fed Reg Supporting Statement |
| Author | STPI |
| Last Modified By | genevieve r dealmeida |
| File Modified | 2012-09-13 |
| File Created | 2012-09-13 |