Nsf 2011 Survey Of Principal Investigators And Reviewers

NSF 2011 Survey of Principal Investigators and Reviewers

Introduction

Thank you for participating in our survey.

The National Science Board (NSB) is currently undertaking a review of the two merit review criteria (Intellectual Merit and Broader Impacts). As part of that review, the NSB Task Force on Merit Review has contracted with SRI International to assist in gathering and analyzing input from various stakeholders on a number of issues related to the two criteria. These issues include how the criteria are interpreted by both external communities and internal NSF staff, as well as how the criteria are used in the preparation and review of proposals, and in making funding decisions.

This survey is being sent to a random sample of individuals who submitted proposals to the NSF that were awarded or declined during 2009 and 2010 and/or served as a proposal reviewer during that same period.

Your participation in this study is voluntary. You may choose not to provide information that you feel is privileged. There will be no individual attribution to any survey response. SRI as the survey administrator will maintain the confidentiality of all respondents. Any survey data provided to anyone outside of SRI, including NSF or the NSB, will be purged of information that could be used to identify individual responses. Please note:

• This survey contains both structured and open-ended questions; it should take about 15 – 30 minutes of your time to complete, depending on your responses to open-ended questions.

• This survey will be open through March xx, 2011.

• When you complete the survey, please click the "SUBMIT" button at the end.

• If you do not complete the entire survey and choose to return to it at a later time, you will be taken to the point where you left off by clicking the button “LOAD UNFINISHED SURVEY.”

• Please click the button “NEXT” to proceed to the survey.

If you have any technical questions about the web survey, please contact Roland Bardon at [email protected], or 703-247-8545. If you have general questions about the study, please contact me at [email protected] or 703-247-8550.

Sincerely,

Jongwon Park,
Study Director
SRI International

Pursuant to 5 CFR 1320.5(b), an agency may not conduct or sponsor, and a person is not required to respond to an information collection unless it displays a valid OMB control number. The OMB control number for this collection is 3145-0157. Public reporting burden for this collection of information is estimated to average 15-30 minutes per response, including the time for reviewing instructions. Send comments regarding this burden estimate and any other aspect of this collection of information, including suggestions for reducing this burden, to: Reports Clearance Officer, Facilities and Operations Branch, Division of Administrative Services, National Science Foundation, Arlington, VA 22230.

Principal Investigator – Views on Intellectual Merit and Broader Impacts Criteria

1. Have you ever submitted a proposal to NSF?

[___] Yes

[___] No  Skip to Q 16

2. Has a proposal you submitted to NSF been awarded or declined during the past 2 years (Calendar year 2009 – 2010)?

[___] Yes

[___] No  Skip to Q 16

3. Was your most recent NSF proposal decision an award or a declination?

[___] Declination

[___] Award

4. Do you currently have a proposal that you submitted to NSF under consideration?

[___] Yes

[___] No

5. As a PI, how useful was information you obtained regarding the Intellectual Merit Criterion from each of the following sources in helping you prepare the proposal(s) that you submitted to NSF during the past 2 to 3 years?

	Did not use source – Not applicable	Not at all useful	Somewhat useful	Moderately useful	Very useful
NSF Grant Proposal Guide
Other NSF Resources available on the web
Personal contact with NSF Official/Staff – email, phone, or in person
My University/Institution
Professional Association
Peers
Feedback from NSF on previous proposal(s) I submitted
Other (Please specify below.)

Other sources of information and usefulness

6. As a PI, how useful was information you obtained regarding the Broader Impacts Criterion from each of the following sources in helping you prepare the proposal(s) that you submitted to NSF during the past 2 to 3 years?

	Did not use source – Not applicable	Not at all useful	Somewhat useful	Moderately useful	Very useful
NSF Grant Proposal Guide
Other NSF Resources available on the web
Personal contact with NSF Official/Staff – email, phone, or in person
My University/Institution
Professional Association
Peers
Feedback from NSF on previous proposal(s) I submitted
Other (Please specify below.)

Other sources of information and usefulness

7. How useful was the most recent proposal feedback, if any, that you received during the past 2 years related to the Intellectual Merit criterion and the Broader Impacts criterion?

	No feedback received	Not at all useful	Somewhat useful	Moderately useful	Very useful
Intellectual Merit
Broader Impacts

8. Based on the review(s) of NSF proposal(s) that you received a decision on during the past 2 year period, how would you rate the overall level of understanding exhibited by reviewers of the Intellectual Merit and Broader Impacts criteria?

	Very high level	High level	Moderate level	Low level	Very low level	No basis to judge
Intellectual Merit Criterion Understanding
Broader Impacts Criterion Understanding

9. Based on your experiences submitting proposals to NSF during the past 2 to 3 years, how much weight did reviewers place on the Intellectual Merit Criterion compared to the Broader Impacts Criterion in the NSF review process?

[__] Much more weight on Intellectual Merit

[___] More weight on Intellectual Merit

[___] Somewhat more weight on Intellectual Merit

[___] Equal weight on both

[___] Somewhat more weight on Broader Impacts

[___] More weight on Broader Impacts

[___] Much more weight on Broader Impacts

[___] No basis to judge

[___] Other (Please explain.)

10. In your opinion, how much weight should reviewers typically place on the Intellectual Merit Criterion compared to the Broader Impacts Criterion?

[___] Much more weight on Intellectual Merit

[___] Somewhat more weight on Intellectual Merit

[___] Equal weight on both

[___] Somewhat more weight on Broader Impacts

[___] Much more weight on Broader Impacts

[___] No basis to judge

[___] Other (Please specify.)

11. In your opinion, should your institution play a greater or lesser role in supporting the portion of PIs’ proposals designed to satisfy the Intellectual Merit and Broader Impacts criteria?

	Much greater	Somewhat greater	Stay the same	Somewhat less	Much less	No basis to judge
Intellectual Merit
Broader Impacts

12. To what extent did the Broader Impacts activities in your most recent NSF proposal address each of the following?

	Little, or no extent	Some extent	Moderate extent	Great extent	Very great extent	No basis to judge
Increased economic competitiveness of the United States.
Development of a globally competitive STEM (Science, Technology, Engineering, and Mathematics) workforce.
Increased participation of women and underrepresented minorities in STEM.
Increased partnerships between academia and industry.
Improved pre-K–12 STEM education and teacher development.
Improved undergraduate STEM education.
Increased public scientific literacy.
Increased national security.

13. What portion, if any, of the Broader Impacts activities specified in your most recent NSF proposal went beyond those activities associated with doing the research and reporting the results to other researchers?

[___] All or almost all

[___] Most

[___] About half

[___] Some

[___] None

14. In your most recent NSF proposal did your budget include costs associated with activities you had identified as related to Broader Impacts criterion?

[___] Yes

[___] No

15. What suggestions, if any, do you have for ways your university/institution could do more to support PIs in their efforts to meet the NSF’s Intellectual Merit and Broader Impacts criteria?

Support of Intellectual Merit

Support of Broader Impact

Reviewer – Views on Intellectual Merit and Broader Impacts Criteria

16. During the past 2 years have you served as an NSF reviewer on a review panel or as an individual reviewer outside the panel system on an ad hoc basis?

[___] I have served on a review panel only

[___] I have served as an individual reviewer on ad hoc basis only

[___] I have served as both panel and ad hoc reviewer

[___] I not served as an NSF reviewer - Skip to Background Q 31

17. As a reviewer, how useful was information you obtained regarding the Intellectual Merit Criterion from each of the following sources in assessing the proposals you reviewed during the past 2 years?

	Did not use source – Not applicable	Not at all useful	Somewhat useful	Moderately useful	Very useful
NSF Grant Proposal Guide
Other NSF Resources available on the web
NSF Program Officer
Other NSF Staff
My University/Institution
Professional Association
Peers
Feedback from NSF on proposal(s) I submitted
Other (Please specify below.)

Other sources of information and usefulness

18. As a reviewer, how useful was information you obtained regarding the Broader Impacts Criterion from each of the following sources in assessing the proposals you reviewed during the past 2 years?

	Did not use source – Not applicable	Not at all useful	Somewhat useful	Moderately useful	Very useful
NSF Grant Proposal Guide
Other NSF Resources available on the web
NSF Program Officer
Other NSF Staff
My University/Institution
Professional Association
Peers
Feedback from NSF on proposal(s) I submitted
Other (Please specify below.)

Other sources of information and usefulness

Skip instruction here - Skip Q 19 if answered Q 16 During the past 2 years have you served as an NSF reviewer on a review panel or as an individual reviewer outside the panel system on an ad hoc basis? As – “I have served as an individual reviewer on ad hoc basis only “

19. Based on your experiences as an NSF review panel member during the past 2 years, how much weight did other reviewers typically place on the Intellectual Merit Criterion compared to the Broader Impacts Criterion?

[__] Much more weight on Intellectual Merit

[___] More weight on Intellectual Merit

[___] Somewhat more weight n Intellectual Merit

[___] Equal weight on both

[___] Somewhat more weight on Broader Impacts

[___] More weight on Broader Impacts

[___] Much more weight on Broader Impacts

[___] No basis to judge

[___] Other (Please specify.)

20. In your opinion, how much weight should Reviewers place on the Intellectual Merit Criterion compared to the Broader Impacts Criterion?

[___] Much more weight on Intellectual Merit

[___] More weight on Intellectual Merit

[___] Somewhat more weight on Intellectual Merit

[___] Equal weight on both

[___] Somewhat more weight on Broader Impacts

[___] More weight on Broader Impacts

[___] Much more weight on Broader Impacts

[___] No basis to judge

[___] Other (Please specify.)

21. How many of the proposals that you reviewed during the past 2 years contained specific Broader Impacts goals and activities that went beyond those activities associated with doing the research and reporting the results to other researchers?

[___] None

[___] Some

[___] About half

[___] Most

[___] All or almost all

[___] Do not recall

22. How many of the proposals that you reviewed during the past 2 years included costs in the budget to support goals or activities the PI had identified as related to Broader Impacts?

[___] None

[___] Some

[___] About half

[___] Most

[___] All or almost all

[___] Do not recall

PI and Reviewers – Views

23. In the Grant Proposal Guide, NSF provides the following list of potential considerations for the Intellectual Merit criterion:

• How important is the proposed activity to advancing knowledge and understanding within its own field or across different fields?
• How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment on the quality of prior work.) 
• To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts? 
• How well conceived and organized is the proposed activity? 
• Is there sufficient access to resources? 

How would you rate this list as guidance for PIs in formulating proposals? -- and for reviewers in assessing proposals?

Please choose the appropriate response for each item:

	Excellent	Good	Fair	Poor	Very poor	No basis to judge
For PIs in formulating proposals
For reviewers in assessing proposals

24. In the Grant Proposal Guide NSF also provides the following list of potential considerations for the Broader Impacts criterion: 

• How well does the activity advance discovery and understanding while promoting teaching, training, and learning? 
• How well does the proposed activity broaden the participation of underrepresented groups (such as gender, ethnicity, disability, geographic, etc.)? 
• To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships? 
• Will the results be disseminated broadly to enhance scientific and technological understanding? 
• What may be the benefits of the proposed activity to society? 

How would you rate this list as guidance for PIs in formulating proposals? -- and for reviewers in assessing proposals?

Please choose the appropriate response for each item:

	Excellent	Good	Fair	Poor	Very poor	No basis to judge
For PIs in formulating proposals
For reviewers in assessing proposals

25. What suggestions, if any, would you offer to improve the guidance NSF provides to PIs and reviewers in Grant Proposal Guide regarding the Intellectual Merit and Broader Impacts criteria including revisions or additions to the lists of potential considerations identified in the previous questions?

Suggestions for improving guidance on Intellectual Merit criterion including the list of potential considerations

Suggestions for improving guidance on Broader Impacts criterion including the list of potential considerations

27. In your opinion, should NSF do more or less than it is currently doing to assess whether or not the goals of Intellectual Merit and Broader Impacts were realized in the completed research it funded?

	Much more	Somewhat more	About the same	Somewhat less	Much less	No basis to judge
Intellectual Merit
Broader Impacts

28. What suggestions, if any, do you have for ways NSF could do more to assess the extent to which thewhichthe the goals of Intellectual Merit and Broader Impacts of the completed research it funded were realized?
    
    

Ways to assess whether or not Intellectual Merit goals of funded research were realized

Ways to assess whether or not Broader Impact goals of funded research were realized

29. What do you view as the major strengths, if any, of the Intellectual Merit and Broader Impacts criteria?

Major Strengths of Intellectual Merit Criterion

Major Strengths of Broader Impacts Criterion

30. What do you view as the major weaknesses, if any, of NSF’s Intellectual Merit and Broader Impacts criteria?

Major Weaknesses of Intellectual Merit Criterion

Major Weaknesses of Broader Impacts Criterion

31. If you have any additional comments including suggested improvements to NSF’s Merit Review Criteria or related issues, please provide them below.

Additional Comments

BACKGROUND

The following demographic questions are asked for statistical purposes. Your responses are voluntary.

32. What is your Ethnicity?

[___] Hispanic or Latino

[___] Not Hispanic or Latino

33. What is your race? (Select one or more.)

[__] American Indian or Alaska Native

[__] Asian

[__] Black or African American

[__] Native Hawaiian or Other Pacific Islander

[__] White

34. What is your gender?

[__] Female

[__] Male

35. What is your current disability status? (Select one or more.)

[__] None

[__] Hearing impairment not corrected with hearing aid

[__] Visual impairment not corrected with glasses

[__] Mobility/Orthopedic impairment

[__] Other (Please specify)

36. Which of the following best describes your citizenship and current residency status?

[___] US citizen residing in the US

[___] US citizen residing outside of the US

[___] Not US citizen residing in the US

[___] Not US citizen residing outside of the US

37. How many years ago did you receive your highest terminal professional degree?

[__] Less than 1 year

[__] 1 – 3 years

[__ ] 3 – 5 years

[__] 5 – 10 years

[__] 10 – 15 years

[__] More than 15 years

[__] Not applicable

28