Attachment A:
Data Collection Instrument
[Survey Introduction]
Thank you for your participation. This survey is designed to help NSF understand the factors that influence the number of NSF proposal submissions. Your responses will help NSF improve its service to the community of proposers and reviewers.
There will be no individual attribution to any survey response.
This survey should take approximately 15 minutes to complete.
Please contact xxxx at [email protected] with any technical or administrative questions about the survey.
Paperwork
Burden Statement
According
to the Paperwork Reduction Act of 1995, no persons are required to
respond to a collection of information unless it displays a valid OMB
control number. The valid OMB control number for this information
collection is 3145-0215. The time required to complete this voluntary
information collected is estimated to average 15 minutes per
response, including the time to review instructions, search existing
data resources, gather the data needed, and complete and review the
information collection. If you have any comments or concerns about
the contents or the status of your individual submission of this
questionnaire, NSF Reports Clearance Officer, 4201 Wilson Boulevard,
Room 1265, Arlington, VA 22230.
[Page Introduction]
This survey consists of two sections. The first asks about your experiences as someone who has reviewed proposals for NSF (if applicable), and the second asks about your experiences as someone who as submitted proposals to NSF (if applicable).
(MASTER FILTER): During the past 3 years, please select the group below that best describes your relationship with NSF:
Proposer (PI) |
Reviewer |
Both PI and Reviewer |
Neither |
<If answer is ‘Neither’, apologize for sending the survey in error and exit.>
Skip logic patterns (involvement with pilot is not determined at this stage):
PI only. After 2, skip to 21
Reviewer and Both. After 2, continue to 3.
Within the past 3 years, with which NSF Directorate(s) have your scholarly activities most closely affiliated? (Note: If your work aligns with more than one NSF Directorate, select up to three Directorates in the drop-down menus below.)
[Set of 3: Progressive drop-down menu of NSF directorates, sub-directorates]
EXPERIENCES AS A REVIEWER
<Here if answered ‘Reviewer’ or ‘Both’ to question 1>
[REVIEWER WORKLOAD]
How many reviews of proposals have you written in the past 3 years:
For NSF:
0
1-3
4-6
7-10
11-15
17-20
21-30
31-40
Over 40
For Other Organizations:
0
1-3
4-6
7-10
11-15
17-20
21-30
31-40
Over 40
For the following questions, please refer to the panelist descriptions provided below.
An ad hoc reviewer is someone who submits a written review of a proposal but does not participate in a discussion of the proposal with other reviewers.
A panelist, or panel reviewer, is someone who participates in a discussion of a proposal (usually more than one proposal) with other reviewers.
There are two types of panelists:
A remote panelist is someone who participates in the panel discussion via telephone, video-conference, web-based virtual meeting technology, or similar.
A face-to-face panelist is someone who gathers with other reviewers at a common location (often NSF) to discuss proposals.
During the past 12 months, have you declined to:
Yes |
No |
Serve as an ad hoc reviewer for NSF
Serve as a face-to-face panelist on an NSF review panel
Serve as a remote panelist on an NSF review panel
Show/Hide: If “yes” to any of above:
On average, to what extent did the following factors influence your decision to decline NSF's review request?
To a Great Extent |
To a Moderate Extent |
To a Small Extent |
To No Extent |
Proposal or program were not relevant to my professional interests
Lack of time
Conflict of interest
Too many NSF review requests
Competing professional pressures (including teaching, organizational administration service, etc.)
Dissatisfaction with the proposal review process
Increasing commitments to other funding agencies
[Make visible only if relevant answer on Q4] Unable to travel to a face-to-face panel
[Make visible only if relevant answer on Q4] Dislike participating in discussions over phone, video-conference, or web-based meeting technology
Please estimate the average amount of time that you spend reading, writing, and submitting a review for a typical NSF proposal
Less than 30 minutes
Between 30 minutes and an hour
Between an hour and 2 hours
Between 2 hours and 3 hours
Between 3 hours and 5 hours
Between 5 hours and 10 hours
More than 10 hours
When do you typically read and write reviews of NSF proposals?
During your normal work-day
Mainly outside of your normal working hours
Both during the work-day and outside your normal working hours
How does your institution view participating as a reviewer (for NSF or other agencies)?
a. My institution considers participating as a reviewer to fall within the scope of my normal work duties.
b. My institution considers participating as a reviewer to fall outside the scope of my normal work duties.
c. I am unsure whether my institution considers participating as a reviewer to fall within or outside the scope of my normal work duties.
Greatly Increased |
Somewhat Increased |
Stayed the same |
Somewhat Decreased |
Greatly Decreased |
The time you are able to devote to each review
The thoroughness you provide to each review
NSF is interested in knowing your panel experiences. NSF holds three types of review panels:
Face-to-face panels. In these all panelists gather at the same location to discuss proposals.
Wholly virtual panels. In these, all panelists participate via telephone, video-conference, web-based virtual meeting technology, or similar.
Hybrid panels. In these, some panelists gather at a common location and others “join” them via a remote connection such as a telephone, video-conference, web-based virtual meeting technology, or similar.
[WHOLLY VIRTUAL PANEL FILTER] Have you participated in a wholly virtual NSF proposal review panel?
Yes continue to 10
No skip to 11
Which of the following technologies have you used in NSF virtual panels? (Check all that apply)
Teleconferencing
Web-based virtual meeting software (e.g. WebEx, BlueJeans, etc.)
Video-conferencing, whether web-based or otherwise (e.g. Skype, iChat, etc.)
Virtual worlds (e.g., Second Life)
For NSF, I have served as a reviewer:
Only on virtual panel(s) Skip to 16
In both virtual panels and face-to-face panels continue to 12
Compare your experience as a virtual panelist to your experience as a face-to-face panelist on the following dimensions.
Significantly Better in Virtual Panel format |
Somewhat in Virtual Panel format |
About the same |
Somewhat Better in Face-to-Face Panel format |
Significantly Better in Face-to-Face Panel format |
Quality of panel briefing/training
Quality of group discussions
Quality of the panel summaries
Quality of interpersonal interaction among panel members
Quality of interaction with NSF staff
Compare your experience as a virtual panelist to your experience as a face-to-face panelist on the following dimensions.
Significantly More in Virtual Panel format |
Somewhat More in Virtual Panel format |
About the same |
Somewhat More in Face-to-Face Panel format |
Significantly More in Face-to-Face Panel format |
Overall time spent preparing for panel
Overall time commitment
Time spent on preparing reviews
Average amount of time spent discussing each proposal
Number of proposals discussed by the panel
Overall satisfaction
Which of the following best describes why you participated in virtual panels?
I have only been invited to participate in virtual panels. Skip to 16
I have declined to participate in traditional face-to-face panels in favor of virtual panels. continue to 15
I am equally likely to participate in virtual panels and traditional face-to-face panels. Skip to 16
Which of the following were factors in your decision to decline participation in traditional face-to-face panels in favor of virtual panels? (Select all that apply)
I am unable to travel.
Travel arrangements for traditional face-to-face panels are too cumbersome.
Scheduling time away from other commitments is too difficult.
The time commitment for traditional face-to-face panels is too great.
I prefer interacting with other co-panelists in a virtual capacity.
Other (please describe):
In your opinion, what could NSF do to improve the experience of serving on both virtual and traditional panels? (Select all that apply)
Reduce required time commitment
Reduce proposal volume
Make the panel schedule more flexible
Facilitate more interaction among co-panelists
Provide additional technical support for panelists
Provide additional training opportunities for panelists
Integrate virtual meeting technology and the FastLane Interactive Panel System so that there is no need to run two applications simultaneously.
Other (please describe):
IF “Proposer” or “Both” are selected for 1
EXPERIENCES AS A PRINCIPAL INVESTIGATOR
NSF is interested in the factors that influence your decision to seek funding from NSF or other sources. For the purposes of this survey, please answer the following questions based on your experience as a principal investigator (PI), not on any experience that you may have had as a co-Principal Investigator (co-PI).
Beyond the goal of making contributions to your area of science, to what extent do the following factors motivate you to submit research proposals to any funding source?
To a Great Extent |
To a Moderate Extent |
To a Small Extent |
To No Extent |
Building/maintaining a record of submitting proposals for academic tenure and/or promotion
Contributing to my employing organization's research status/reputation
Securing funding to pay for my own salary
Supplementing my salary
Being able to continue to pay the salaries of individuals who currently work with me in a professional capacity (e.g. post-doctoral associates, technicians, lab managers, etc.)
Being able to continue to pay the stipends of graduate students who currently work with me
To enable me to involve students (graduate, undergraduate or high school) in research
To pay for the acquisition, development, maintenance, or operation of laboratory equipment and instrumentation.
For all references to success rates in this survey, please use the following definition:
NSF currently defines success rate as the number of awards granted in a given year divided by the number of proposals reviewed that year.
To what extent did the following factors influence your decision to submit to NSF during the past 3 years?
To a Great Extent |
To a Moderate Extent |
To a Small Extent |
To No Extent |
Decreased funding available from other sources
Better chance of funding at NSF than other agencies
Pressure to submit proposals for tenure and/or promotion
Pressure to obtain grants for tenure and/or promotion
Pressure to build and maintain research facilities, centers or programs
NSF is the major source of funding for my area of research
The NSF budget in my area of research has increased
Interesting and relevant new funding opportunities
Opportunities for funding inter-, cross-, or multidisciplinary research
Opportunities for funding collaborative research
Encouragement from NSF staff
Reflecting on your most recent proposal submitted to NSF, what do you think the success rate was for the program you submitted to?
5% or Less
6-10%
11-20%
21-30%
31-40%
Over 40%
I'm Not Sure
Over the next 5 years, I view NSF as the prime source of potential funding for the following percentage of my research:
Less than 10%
10-25%
26-50%
51-75%
76-100%
Have you applied for awards or grants from federal institutions other than the National Science Foundation?
Yes show all bullets in 22
No show last 2 bullets in 22, then skip to 24
In the last 3 years, how often have you...
Frequently |
Seldom |
Never |
Submitted very similar proposals simultaneously to NSF and other funding agencies
Submitted a proposal to NSF that was declined by another agency
After an NSF proposal has been declined, submitted a revised version of that NSF proposal to the same NSF program / division (except those cases in which you were explicitly invited to resubmit a revision)
After an NSF proposal has been declined, submitted a revised version of that NSF proposal to other NSF programs / divisions
Compared to other federal agencies, how do you perceive the level of competition for research grants in your area of research at NSF?
More intense at NSF than at other federal agencies
The same at NSF as at other federal agencies
Less intense at NSF than at other federal agencies
In general, after how many declines of a proposed project would you...
Stop submitting the project to any agency
1
2-3
4-6
7-9
10 or more
Stop submitting the project anywhere within NSF
1
2-3
4-6
7-9
10 or more
Stop submitting the project to a particular NSF program
1
2-3
4-6
7-9
10 or more
What is the expected success rate at or below which you would you would no longer consider applying to NSF?
5%
10%
20%
30%
40%
I would always consider applying
To what extent do the reviews that accompany the declination of one of your NSF proposals...
To a Great Extent |
To a Moderate Extent |
To a Small Extent |
To No Extent |
Improve your understanding of the proposal process
Provide useful information for revising and improving your next proposal
Influence you to submit to another funding agency
Negatively impact your career
How long have you been submitting proposals to NSF?
Less than 3 years Skip to 29
3 years or more
How have the following changed over the past 3 years?
Greatly Increased |
Somewhat Increased |
Stayed the same |
Somewhat Decreased |
Greatly Decreased |
The overall quality of feedback in the written reviews of your proposals
The overall quality of feedback from NSF staff about your proposals
The timeliness of the decision to award or not award funding
The timeliness of responses by NSF staff to your inquiries
The quality of your interaction with NSF staff
Which of the following best describes the NSF program to which you most frequently submit proposals?
The NSF program to which I most frequently submit proposals has no deadlines or target dates and accepts proposals at any time.
The NSF program to which I most frequently submit proposals has two or more deadlines or target dates each year.
The NSF program to which I most frequently submit proposals has only one deadline or target date each year.
There is no NSF program to which I typically submit proposals.
Within the past 3 years, on average, I have submitted the following number of proposals per year to NSF:
An average of fewer than one per year
Between 1 and 1.5 per year
Between 1.5 and 2.5 per year
Between 2.5 and 3.5 per year
Between 3.5 and 5.5 per year
More than 5.5 per year
For the following questions, please refer to the proposals you have submitted to the NSF within the past 3 years for which you have received an award or decline decision.
[PI SATISFACTION]
How satisfied were you with...
Very Satisfied |
Somewhat Satisfied |
Neutral |
Somewhat Dissatisfied |
Very Dissatisfied |
The quality of the information NSF provided during the proposal submission process (i.e., FastLane, FAQs, web site content)
The timeliness of the decision to award or not award funding
Your interaction with NSF staff
[PI WORKLOAD]
Compared to other agencies' proposal submission systems, how much effort does it take to complete a proposal in the required format and submit it to NSF?
More Effort
Nearly the Same Effort
Less Effort
Not Applicable
Please estimate the average amount of time that you spend individually preparing a full proposal to NSF (Note: If you sometimes revise and resubmit a proposal after it is declined, please count the different versions of your proposal as separate proposals when estimating this average):
Less than 10 hours
Between 10 and 20 hours
Between 20 and 40 hours
Between 40 and 60 hours
Between 60 and 80 hours
Between 80 and 100 hours
Between 100 and 120 hours
Based on your experience with NSF, to what extent do you agree with the following statements?
Strongly Agree |
Agree |
Disagree |
Strongly Disagree |
All proposers are treated fairly
Reviews are thorough
Reviews are technically sound
The background information regarding the outcomes of the competition was of high quality.
The comments I received from my Program Officer were of high quality.
The conversations (email, phone, face-to-face) I had with my Program Officer were of high quality.
The Panel Summary was of high quality
The merit review process provides feedback that I can use to improve my future proposals
The merit review process provides feedback that increases my likelihood of future success
Time spent writing proposals
[ALL RESPONDENTS]
35. This customer service survey has asked about your experiences with NSF’s Merit Review process. In your opinion, improving which one of the following factors in that process would have the most significant effect in fostering the progress of science? (Select one.)
Timeliness of decisions about, and responsiveness to, proposals by NSF staff
Quality of feedback to PIs in the form of review comments and panel summaries
Quality of PI conversations with, and written comments from, program directors
Quality of information available during proposal submission
Quality of the review process from the perspective of a reviewer
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Meg Trucano |
File Modified | 0000-00-00 |
File Created | 2021-01-28 |