May 4, 2009
Name: G. Stephane Philogene, Ph.D.
Address: National Institutes of Health
31 Center Drive, Room B1C32
Bethesda, MD 20892-2027
Telephone: 301.402.3902
Fax: 301.402.1150
Email: [email protected]
B. Collection of Information Employing Statistical Methods 1
B.1 Respondent Universe and Sampling Methods 1
B. 2. Procedures for the Collection of Information 1
B.3. Methods to Maximize Response Rates 3
B.4. Test of Procedures or Methods to be Undertaken 4
B.5. Individuals Consulted on Statistical Aspects and Individual Collecting and/or Analyzing Data 4
List of Attachments:
Attachment 1: NIA Program Description and Requirements
Attachment 2a: New Investigator Award Applicant Survey Instrument
Attachment 2b: Extramural Reviewers Informal Discussion Guide
Attachment 3: Privacy Procedures of Contractor
Attachment 4: Respondent Instructions
Attachment 5: Introductory & Follow-up Instructions
This request seeks approval for OMB clearance for the NIH Director’s New Innovator Award (NIA) Program Process Evaluation. The data collection consists of one survey to be completed by a sample of participants (applicants and awardees). The information gathered from these surveys will document the NIA program operations and will be used as a guide to the program officers in their future strategic and management decisions.
A census will be conducted of all NIA recipients for the FY 2007, FY 2008, and FY 2009 competitions. The approximate total number of NIA participants to be surveyed over the three years is approximately 1710 applicants and 276 external evaluators. A census is requested rather than a sample because of the program’s wide diversity of awards. NIA awards have dealt with a wide range of scientific disciplines and methods. Sampling would result in the number of respondents from important sub-groups being too small to permit comparative analyses, as well as preclude analyses of factors that influence various outcomes.
At the outset of the survey process, an email from NIA officers will contact all participants to describe the purpose of the evaluation, the procedures being used, and requesting that they participate in the survey. They will also introduce the Science and Technology Policy Institute. To review all communications with the participants please consult Attachments 4 and 5 to this document.
The same week, researchers from the Science and Technology Policy Institute will contact all participants with information on how they should locate the survey on the Internet, and procedures for completing the survey. Respondents will be allowed 3 weeks to complete the survey. In order to facilitate completion, individuals may either fill out the survey in one sitting, or they may pause at any time, and return later to finalize the survey. The survey will be organized by topics and participants can move from section to section via a central navigation interface.
Once the individual surveys are underway, the Science and Technology Policy Institute will monitor the status of the submissions (Never Logged In, Not Started, In Progress, and Complete). After two weeks, the first reminder will be sent to individuals who had never logged in. Individuals who have initiated but have not yet completed the survey will receive a note encouraging them to complete their survey; hard-copy of the survey is mailed to participants who have requested it in paper form. All persons who have completed the survey will be sent a thank you note. The second reminder email will be sent two weeks after the first (and about four weeks after the start of the survey process). All new respondents will be thanked at that time, and the persons who had started the survey will be encouraged to complete it. The survey will be closed at the end of week 8. The collection timeline for the survey is outlined in Table B.2.1
To collect data for the Extramural Reviewers specifically, we will perform phone interviews using an interview protocol. To select candidates for a phone interview, a stratified purposeful sampling strategy will be used. All evaluators are invited to participate; but, based on the responses, a convenience sample will be calculated to fill desired strata for analysis; by stratifying the sample, and thus the data collected will be as representative as possible. A series of phone interviews are then scheduled by the Science and Technology Policy Institute via email throughout eight weeks to meet the stratified sample targets. Thank you notes are emailed upon completion of each interview. The collection timeline for the interview is also outlined in Table B.2.1.
The whole process from the beginning to end is expected to take about 8 weeks (Table B.2.1). Those who do not wish to participate will be classified as non-respondents. When the survey and interviews are completed, a non-respondent analysis by demographics and other characteristics will be conducted.
B.2.1 Estimated Collection Procedures |
||
Week |
Applicant Survey |
Extramural Reviewers’ Interview |
1 |
Email introduction by NIA; survey begins (deadline end of week 3) |
Email introduction by NIA; scheduling of interviews begins |
3 |
First reminder to non-respondents |
Interviews; thank you note to respondents. Scheduling of interviews. |
4 |
Encouragement to complete the survey; mailing of hard-copy survey; thank you note to respondents |
Interviews; thank you note to respondents. Scheduling of remaining interviews based on purposeful sampling. |
5 |
Second reminder to non-respondents |
Interviews; thank you note to respondents. |
6 to 8 |
Encouragement to complete the survey; mailing of hard-copy survey; thank you note to respondents |
Complete interviews based on purposeful sampling. |
8 |
Survey ends; thank you note to respondents |
Interview ends; thank you note to respondents. |
Staff at the Science and Technology Policy Institute will perform all analyses.
This study is being conducted with the aim of describing or counting the population characteristics, activities, or outcome of interest. The analyses employed for this purpose will be descriptive, and will involve measures or estimates of central tendency (mean, median, or mode), or of dispersion (variance or standard deviation) in the data, as well as measures of the frequency and range of a variable. An example of such statistic is calculating the mean age of applicants, or determining what fraction of these applicants are women.
Descriptive statistical techniques also include stratified analyses and cross-tabulations. In a stratified analysis, the data will be divided into groups, such as age, gender, affiliation, etc., in order to gain a better understanding of the distribution of the data, and to see if any trends across strata may exist where this is relevant. For example, we can examine whether there are fewer women among clinical research than among other research areas.
An evaluation of the distribution of data in which there are at least two factors of interest has traditionally been made through a significance test. A significance test is usually applied when the goal of a study is to examine relationships between different factors of interest, rather than simply to describe a population. Statistical tests are based upon an underlying set of assumptions that include the stipulation of the appropriate probability distribution (such as the normal distribution, the binomial, or the hypergeometric). In certain cases, we have hypotheses concerning the distribution of data that will guide our undertaking of particular analyses.
Where there are several factors of interest that need to be considered simultaneously, more advanced statistical techniques may be used. These involve multivariate methods. One of the first steps involved in a multivariate analysis is the selection of an appropriate model. The initial descriptive analysis of the data provides information which is useful, and the questions that are posed by the study will also be a determining factor in model selection. Among the frequently used types of multivariate analysis are ordinary least squares, logistic, or Poisson regression. The multivariate model typically includes a dependent variable or outcome, and a set of independent variables that serve to explain or predict the outcome. When independent variables (such as gender or age) are used in a multivariate model, some decision must be made as to how they will be entered. Gender has only two categories, but age or education may be grouped in various ways, or may be entered into a model as continuous variables. For example, we can incorporate several variables in a model (availability of additional funding, gender, and seniority) and try to determine whether these factors can accurately predict the outcome of the selection process. Once the analyses are complete, the Science and Technology Policy Institute will prepare a thorough documentation of the findings and conclusions, and suggest specific recommendations regarding program activities, accomplishments, and administration.
Prior year collections for the New Director’s Pioneer Award (NDPA) indicate high interest by participants in responding to the survey and participating in the interview collections. For example, in NDPA FY 2006, 71% of all participants responded to the survey and the minimum target response rate was above 68% for all subgroups (in some cases, 100%, such as NDPA selected interviewees and awardees). Therefore, current methods appear to be generally effective in maximizing response rates; these methods include but are not limited to:
Monitor web-page responses, and contact non-respondents by email to both solicit their cooperation, and to enhance the ultimate response rates. These procedures are described in detail in section B.2. In rare cases, when participants do not have access to email, or if their web browsers cannot handle the survey, a hard copy of the instrument will be mailed.
As discussed elsewhere, respondents will be able to move freely from section to section and to log out and finish the survey at a later time. When logging out respondents will be given an estimate of future effort (e.g. "survey 50% complete"); if a respondent wishes to submit an incomplete survey, they will be able to do so at any time.
Up-to-date information provided by NIH. Since participant information is current and recent, it is less likely that an email address is no longer valid. But if the email is out-dated, web searches will be conducted in an attempt to update the contact information. Previous collections have revealed that because PIs are based in top US universities, their email contacts are often available online.
Participants contacted in previous years showed great interest for the program and desire to improve it. The survey is designed to decrease burden by pre-loading available NIH information and skipping questions that do not pertain to known subgroups. Although respondents will be allowed to submit unfinished surveys, answers to few carefully selected questions will be strongly encouraged. We anticipate that this flexibility will contribute to higher response rates, as has been in past collections.
Procedures have been tested in a previously approved collection for NDPA for FY 2004-2008 where procedures were validated by real time benchmarks that provide accurate estimates of completion of survey and interviews used to calculate estimates for this package. Methods outlined in B.3 have been applied consistently during the previous three NDPA collections, resulting in high returns and minimum burden on participants. A combination of applicant high interest in participating in the survey, streamlined survey design, and stratified sampling for External Reviewer’s interviews, has yielded survey return rates on or above 71% coupled with a general satisfaction by participants with the collection goals and implementation. The lessons learned from the NDPA survey will be used for the NIA survey.
Several individuals from Science and Technology Policy Institute participated in the development of the study plan. The team included persons with knowledge of statistical methods, experience in evaluation of science programs, and expertise in scientific research. Bhavya Lal is Project Director of the NIA process evaluation and the point of contact for this project; she has extensive evaluation experience with NIH and other Federal agencies. She can be contacted at:
Bhavya Lal
Science & Technology Policy Institute
1899 Pennsylvania Avenue NW, Suite 520
Washington DC 20006
Telephone: (202) 419-3720
Facsimile: (202) 223-1695
E-mail: [email protected]
Follow-up emails and telephone calls will be conducted by qualified researchers at the Science and Technology Policy Institute. The project team from the Science and Technology Policy Institute can be reached at (202) 419-3720.
| File Type | application/msword |
| File Title | NDPA Proc Eval 30 day Fed Reg Supporting Statement |
| Author | STPI |
| Last Modified By | curriem |
| File Modified | 2009-05-08 |
| File Created | 2009-01-29 |