Satisfaction of Reviewers and Principal Investigators with NSF's Merit Review Process

Attachment A:

Data Collection Instrument

[Survey Introduction]

Thank you for your participation. This survey is designed to help NSF understand the factors that influence the number of NSF proposal submissions. Your responses will help NSF improve its service to the community of proposers and reviewers.

There will be no individual attribution to any survey response.

This survey should take approximately 15 minutes to complete.

Please contact xxxx at [email protected] with any technical or administrative questions about the survey.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 3145-0215. The time required to complete this voluntary information collected is estimated to average 15 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments or concerns about the contents or the status of your individual submission of this questionnaire, NSF Reports Clearance Officer, 4201 Wilson Boulevard, Room 1265, Arlington, VA 22230.

[Page Introduction]

This survey consists of two sections. The first asks about your experiences as someone who has reviewed proposals for NSF (if applicable), and the second asks about your experiences as someone who as submitted proposals to NSF (if applicable).

1. (MASTER FILTER): During the past 3 years, please select the group below that best describes your relationship with NSF:

Proposer (PI) 

Reviewer 

Both PI and Reviewer 

Neither 

<If answer is ‘Neither’, apologize for sending the survey in error and exit.>

Skip logic patterns (involvement with pilot is not determined at this stage):

1. PI only.  After 2, skip to 21

2. Reviewer and Both. After 2, continue to 3.

2. Within the past 3 years, with which NSF Directorate(s) have your scholarly activities most closely affiliated? (Note: If your work aligns with more than one NSF Directorate, select up to three Directorates in the drop-down menus below.)

[Set of 3: Progressive drop-down menu of NSF directorates, sub-directorates]

EXPERIENCES AS A REVIEWER

<Here if answered ‘Reviewer’ or ‘Both’ to question 1>

[REVIEWER WORKLOAD]

3. How many reviews of proposals have you written in the past 3 years:

For NSF:

1. 0

2. 1-3

3. 4-6

4. 7-10

5. 11-15

6. 17-20

7. 21-30

8. 31-40

9. Over 40

For Other Organizations:

1. 0

2. 1-3

3. 4-6

4. 7-10

5. 11-15

6. 17-20

7. 21-30

8. 31-40

9. Over 40

For the following questions, please refer to the panelist descriptions provided below.

• An ad hoc reviewer is someone who submits a written review of a proposal but does not participate in a discussion of the proposal with other reviewers.

• A panelist, or panel reviewer, is someone who participates in a discussion of a proposal (usually more than one proposal) with other reviewers.

There are two types of panelists:

• A remote panelist is someone who participates in the panel discussion via telephone, video-conference, web-based virtual meeting technology, or similar.

• A face-to-face panelist is someone who gathers with other reviewers at a common location (often NSF) to discuss proposals.

4. During the past 12 months, have you declined to:

Yes

No

• Serve as an ad hoc reviewer for NSF

• Serve as a face-to-face panelist on an NSF review panel

• Serve as a remote panelist on an NSF review panel

• Show/Hide: If “yes” to any of above:

5. On average, to what extent did the following factors influence your decision to decline NSF's review request?

To a Great Extent

To a Moderate Extent

To a Small Extent

To No Extent

• Proposal or program were not relevant to my professional interests

• Lack of time

• Conflict of interest

• Too many NSF review requests

• Competing professional pressures (including teaching, organizational administration service, etc.)

• Dissatisfaction with the proposal review process

• Increasing commitments to other funding agencies

• [Make visible only if relevant answer on Q4] Unable to travel to a face-to-face panel

• [Make visible only if relevant answer on Q4] Dislike participating in discussions over phone, video-conference, or web-based meeting technology

6. Please estimate the average amount of time that you spend reading, writing, and submitting a review for a typical NSF proposal

1. Less than 30 minutes

2. Between 30 minutes and an hour

3. Between an hour and 2 hours

4. Between 2 hours and 3 hours

5. Between 3 hours and 5 hours

6. Between 5 hours and 10 hours

7. More than 10 hours

7. When do you typically read and write reviews of NSF proposals?

1. During your normal work-day

2. Mainly outside of your normal working hours

3. Both during the work-day and outside your normal working hours

8. How does your institution view participating as a reviewer (for NSF or other agencies)?

a. My institution considers participating as a reviewer to fall within the scope of my normal work duties.

b. My institution considers participating as a reviewer to fall outside the scope of my normal work duties.

c. I am unsure whether my institution considers participating as a reviewer to fall within or outside the scope of my normal work duties.

Greatly Increased

Somewhat Increased

Stayed the same

Somewhat Decreased

Greatly Decreased

• The time you are able to devote to each review

• The thoroughness you provide to each review

NSF is interested in knowing your panel experiences. NSF holds three types of review panels:

• Face-to-face panels. In these all panelists gather at the same location to discuss proposals.

• Wholly virtual panels. In these, all panelists participate via telephone, video-conference, web-based virtual meeting technology, or similar.

• Hybrid panels. In these, some panelists gather at a common location and others “join” them via a remote connection such as a telephone, video-conference, web-based virtual meeting technology, or similar.

9. [WHOLLY VIRTUAL PANEL FILTER] Have you participated in a wholly virtual NSF proposal review panel?

   1. Yes  continue to 10

   2. No  skip to 11

10. Which of the following technologies have you used in NSF virtual panels? (Check all that apply)

    1. Teleconferencing

    2. Web-based virtual meeting software (e.g. WebEx, BlueJeans, etc.)

    3. Video-conferencing, whether web-based or otherwise (e.g. Skype, iChat, etc.)

    4. Virtual worlds (e.g., Second Life)

11. For NSF, I have served as a reviewer:

    1. Only on virtual panel(s)  Skip to 16

    2. In both virtual panels and face-to-face panels  continue to 12

12. Compare your experience as a virtual panelist to your experience as a face-to-face panelist on the following dimensions.

Significantly Better in Virtual Panel format

Somewhat in Virtual Panel format

About the same

Somewhat Better in Face-to-Face Panel format

Significantly Better in Face-to-Face Panel format

• Quality of panel briefing/training

• Quality of group discussions

• Quality of the panel summaries

• Quality of interpersonal interaction among panel members

• Quality of interaction with NSF staff

13. Compare your experience as a virtual panelist to your experience as a face-to-face panelist on the following dimensions.

Significantly More in Virtual Panel format

Somewhat More in Virtual Panel format

About the same

Somewhat More in Face-to-Face Panel format

Significantly More in Face-to-Face Panel format

• Overall time spent preparing for panel

• Overall time commitment

• Time spent on preparing reviews

• Average amount of time spent discussing each proposal

• Number of proposals discussed by the panel

• Overall satisfaction

14. Which of the following best describes why you participated in virtual panels?

    1. I have only been invited to participate in virtual panels.  Skip to 16

    2. I have declined to participate in traditional face-to-face panels in favor of virtual panels.  continue to 15

    3. I am equally likely to participate in virtual panels and traditional face-to-face panels.  Skip to 16

15. Which of the following were factors in your decision to decline participation in traditional face-to-face panels in favor of virtual panels? (Select all that apply)

    1. I am unable to travel.

    2. Travel arrangements for traditional face-to-face panels are too cumbersome.

    3. Scheduling time away from other commitments is too difficult.

    4. The time commitment for traditional face-to-face panels is too great.

    5. I prefer interacting with other co-panelists in a virtual capacity.

    6. Other (please describe):

16. In your opinion, what could NSF do to improve the experience of serving on both virtual and traditional panels? (Select all that apply)

    1. Reduce required time commitment

    2. Reduce proposal volume

    3. Make the panel schedule more flexible

    4. Facilitate more interaction among co-panelists

    5. Provide additional technical support for panelists

    6. Provide additional training opportunities for panelists

    7. Integrate virtual meeting technology and the FastLane Interactive Panel System so that there is no need to run two applications simultaneously.

    8. Other (please describe):

IF “Proposer” or “Both” are selected for 1

EXPERIENCES AS A PRINCIPAL INVESTIGATOR

NSF is interested in the factors that influence your decision to seek funding from NSF or other sources. For the purposes of this survey, please answer the following questions based on your experience as a principal investigator (PI), not on any experience that you may have had as a co-Principal Investigator (co-PI).

17. Beyond the goal of making contributions to your area of science, to what extent do the following factors motivate you to submit research proposals to any funding source?

To a Great Extent

To a Moderate Extent

To a Small Extent

To No Extent

• Building/maintaining a record of submitting proposals for academic tenure and/or promotion

• Contributing to my employing organization's research status/reputation

• Securing funding to pay for my own salary

• Supplementing my salary

• Being able to continue to pay the salaries of individuals who currently work with me in a professional capacity (e.g. post-doctoral associates, technicians, lab managers, etc.)

• Being able to continue to pay the stipends of graduate students who currently work with me

• To enable me to involve students (graduate, undergraduate or high school) in research

• To pay for the acquisition, development, maintenance, or operation of laboratory equipment and instrumentation.

For all references to success rates in this survey, please use the following definition:

NSF currently defines success rate as the number of awards granted in a given year divided by the number of proposals reviewed that year.

18. To what extent did the following factors influence your decision to submit to NSF during the past 3 years?

To a Great Extent

To a Moderate Extent

To a Small Extent

To No Extent

• Decreased funding available from other sources

• Better chance of funding at NSF than other agencies

• Pressure to submit proposals for tenure and/or promotion

• Pressure to obtain grants for tenure and/or promotion

• Pressure to build and maintain research facilities, centers or programs

• NSF is the major source of funding for my area of research

• The NSF budget in my area of research has increased

• Interesting and relevant new funding opportunities

• Opportunities for funding inter-, cross-, or multidisciplinary research

• Opportunities for funding collaborative research

• Encouragement from NSF staff

19. Reflecting on your most recent proposal submitted to NSF, what do you think the success rate was for the program you submitted to?

1. 5% or Less

2. 6-10%

3. 11-20%

4. 21-30%

5. 31-40%

6. Over 40%

7. I'm Not Sure

20. Over the next 5 years, I view NSF as the prime source of potential funding for the following percentage of my research:

1. Less than 10%

2. 10-25%

3. 26-50%

4. 51-75%

5. 76-100%

21. Have you applied for awards or grants from federal institutions other than the National Science Foundation?

1. Yes  show all bullets in 22

2. No  show last 2 bullets in 22, then skip to 24

22. In the last 3 years, how often have you...

Frequently

Seldom

Never

• Submitted very similar proposals simultaneously to NSF and other funding agencies

• Submitted a proposal to NSF that was declined by another agency

• After an NSF proposal has been declined, submitted a revised version of that NSF proposal to the same NSF program / division (except those cases in which you were explicitly invited to resubmit a revision)

• After an NSF proposal has been declined, submitted a revised version of that NSF proposal to other NSF programs / divisions

23. Compared to other federal agencies, how do you perceive the level of competition for research grants in your area of research at NSF?

1. More intense at NSF than at other federal agencies

2. The same at NSF as at other federal agencies

3. Less intense at NSF than at other federal agencies

24. In general, after how many declines of a proposed project would you...

Stop submitting the project to any agency

1. 1

2. 2-3

3. 4-6

4. 7-9

5. 10 or more

Stop submitting the project anywhere within NSF

1. 1

2. 2-3

3. 4-6

4. 7-9

5. 10 or more

Stop submitting the project to a particular NSF program

1. 1

2. 2-3

3. 4-6

4. 7-9

5. 10 or more

25. What is the expected success rate at or below which you would you would no longer consider applying to NSF?

1. 5%

2. 10%

3. 20%

4. 30%

5. 40%

6. I would always consider applying

26. To what extent do the reviews that accompany the declination of one of your NSF proposals...

To a Great Extent

To a Moderate Extent

To a Small Extent

To No Extent

• Improve your understanding of the proposal process

• Provide useful information for revising and improving your next proposal

• Influence you to submit to another funding agency

• Negatively impact your career

27. How long have you been submitting proposals to NSF?

1. Less than 3 years  Skip to 29

2. 3 years or more

28. How have the following changed over the past 3 years?

Greatly Increased

Somewhat Increased

Stayed the same

Somewhat Decreased

Greatly Decreased

• The overall quality of feedback in the written reviews of your proposals

• The overall quality of feedback from NSF staff about your proposals

• The timeliness of the decision to award or not award funding

• The timeliness of responses by NSF staff to your inquiries

• The quality of your interaction with NSF staff

29. Which of the following best describes the NSF program to which you most frequently submit proposals?

1. The NSF program to which I most frequently submit proposals has no deadlines or target dates and accepts proposals at any time.

2. The NSF program to which I most frequently submit proposals has two or more deadlines or target dates each year.

3. The NSF program to which I most frequently submit proposals has only one deadline or target date each year.

4. There is no NSF program to which I typically submit proposals.

30. Within the past 3 years, on average, I have submitted the following number of proposals per year to NSF:

    1. An average of fewer than one per year

    2. Between 1 and 1.5 per year

    3. Between 1.5 and 2.5 per year

    4. Between 2.5 and 3.5 per year

    5. Between 3.5 and 5.5 per year

    6. More than 5.5 per year

For the following questions, please refer to the proposals you have submitted to the NSF within the past 3 years for which you have received an award or decline decision.

[PI SATISFACTION]

31. How satisfied were you with...

Very Satisfied

Somewhat Satisfied

Neutral

Somewhat Dissatisfied

Very Dissatisfied

• The quality of the information NSF provided during the proposal submission process (i.e., FastLane, FAQs, web site content)

• The timeliness of the decision to award or not award funding

• Your interaction with NSF staff

[PI WORKLOAD]

32. Compared to other agencies' proposal submission systems, how much effort does it take to complete a proposal in the required format and submit it to NSF?

1. More Effort

2. Nearly the Same Effort

3. Less Effort

4. Not Applicable

33. Please estimate the average amount of time that you spend individually preparing a full proposal to NSF (Note: If you sometimes revise and resubmit a proposal after it is declined, please count the different versions of your proposal as separate proposals when estimating this average):

1. Less than 10 hours

2. Between 10 and 20 hours

3. Between 20 and 40 hours

4. Between 40 and 60 hours

5. Between 60 and 80 hours

6. Between 80 and 100 hours

7. Between 100 and 120 hours

8. More than 120 hours

34. Based on your experience with NSF, to what extent do you agree with the following statements?

Strongly Agree

Agree

Disagree

Strongly Disagree

• All proposers are treated fairly

• Reviews are thorough

• Reviews are technically sound

• The background information regarding the outcomes of the competition was of high quality.

• The comments I received from my Program Officer were of high quality.

• The conversations (email, phone, face-to-face) I had with my Program Officer were of high quality.

• The Panel Summary was of high quality

• The merit review process provides feedback that I can use to improve my future proposals

• The merit review process provides feedback that increases my likelihood of future success

• Time spent writing proposals

[ALL RESPONDENTS]

35. This customer service survey has asked about your experiences with NSF’s Merit Review process. In your opinion, improving which one of the following factors in that process would have the most significant effect in fostering the progress of science?  (Select one.)

 

• Timeliness of decisions about, and responsiveness to, proposals by NSF staff

• Quality of feedback to PIs in the form of review comments and panel summaries

• Quality of PI conversations with, and written comments from, program directors

• Quality of information available during proposal submission

• Quality of the review process from the perspective of a reviewer