Protocol for Conducting NSF Broader Impacts Focus Groups
Preparation (internal training for focus group facilitators)
When planning and preparing for focus groups, consider the following:
Ensure that you have a stable internet connection and that you are using a headset for best audio quality.
Ensure that you have at least one note taker per group, and that these individuals have the equipment or supplies they need to record or take notes.
Moderators should always have their video on. Note takers can choose to have their video off.
Be prepared to record the session before you begin the introduction and always ask for participants’ consent to record before you press record (Note that we intend to ask permission when recruiting participants but will confirm at the start of each focus group session).
If a participant does not consent to being recorded when you ask at the start of the session, do not record the session. Be sure the note taker knows that there will be no recording and to take extra care to take detailed and diligent notes.
Note-taker should document the focus group details using the provided note-taking template. For example, note-taker should record:
Date and time of the focus group,
The types of participants included in the focus group (reviewers, accepted PIs, declined PIs),
The moderator and note taker, and
Any participants that are late or no-shows.
Review the list of “Moderator Best Practices” at the end of this protocol.
When conducting focus groups, use the protocols according to the following guidelines:
Bolded questions should be asked in all focus groups.
Non-bolded questions are optional and can be used as follow-up questions or probes, as needed.
Text shown in red includes moderator instructions and should not be read aloud.
Introduction (Focus group facilitators will share at the beginning of each session) [6-7 minutes]
The interviewer thanks the participant for joining and confirms the participant can hear them. The moderator introduces her/himself and any others who are playing a working role in the activities (i.e., note-takers). The moderator will provide a high-level background of NSF’s merit review process, including the project’s purpose.
I will begin by sharing a high-level background of NSF’s merit review process and the purpose for this project. NSF program officers and external experts evaluate submitted proposals for two main merit review criteria approved by the National Science Board: (1) Intellectual Merit, referring to the potential to advance knowledge, and (2) Broader Impacts, referring to the potential to contribute to society and achieve specific, desired societal outcomes.
Of critical importance to the Foundation is that its merit review process is implemented in a way that is fair, thorough, competitive, and transparent. However, NSF has found that principal investigators and reviewers lack clarity about the Broader Impacts review criterion, despite NSF’s efforts to provide additional guidance.
This project is designed to understand how NSF interprets and applies the Broader Impacts review criterion across its work. Based on findings from this project, Mathematica will provide NSF with recommendations on ways the Foundation could improve the effectiveness of the Broader Impacts review criterion. This project will produce useful insights to inform ongoing NSF strategies to meet the goals established in Section 526 of the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Reauthorization Act of 2010.
The purpose of this focus group is to understand your perspectives and experiences with NSF merit review, especially as they relate to the Broader Impacts review criterion. In piloting our focus group protocols, we have already heard from staff internal and external to NSF with a wide range of perspectives on Broader Impacts. We expect a diversity of opinions on this topic and are excited to learn about your unique perspectives. Your participation, or non-participation, will not affect your ability to qualify for grants.
We would like to record today’s focus group to ensure we capture your experiences and ideas accurately. Our note taker will also record detailed notes [for PDs only: which we can share with you after the call.] All information you provide is confidential and will not be shared outside the Mathematica project team. We ask that they do not share what others have discussed with people outside of this group. We will delete all recordings at the end of the evaluation in January 2025. Do you consent to being recorded during this focus group? [If Yes: hit Record; If No: continue]
Ground Rules (Focus group moderators will share at the beginning of each session)
Before we start, I wanted to share some guiding principles for our discussion:
First, there are no right or wrong answers! You do not have to agree with everyone else in the discussion if that is not how you really feel. We are interested in your experiences and perspectives. This means you can feel comfortable saying good things as well as critical things. We just want to understand your viewpoints.
Second, we recognize that everyone has a different level of comfort sharing their perspectives. Throughout today’s discussion, you can share your thoughts out loud or by responding in the chat. You are also welcome to follow up with us individually after this discussion to share additional feedback.
Third, my role as moderator will be to guide the discussion. However, you should focus on talking to each other, rather than talking to me.
Fourth, we are recording so we ask that you please talk one at a time.
Fifth, we wanted to confirm if you all are comfortable working on a first name basis during today’s discussion. Please give me a nod if you are comfortable with that.
Finally, we recognize you may have had other involvement or roles with NSF in the past. For today’s discussion, please focus on your experiences as a [reviewer for NSF / as a proposer and PI on NSF-funded projects].
************************************************************************************
Introductions: I would like to start by learning more about each other. Can you each take about 30 seconds to introduce yourself, including your name, where you work, the focus of your work, and how long you’ve been involved with NSF? [call on staff one by one; 6 minutes]
Perception and understanding of Broader Impacts: [20-25 minutes] For the first set of questions, I’d like to understand how you interpret Broader Impacts. According to NSF, “NSF funds scientists and engineers to perform research that advances discovery and innovation. The agency also expects researchers’ work to have broader impacts: the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.”
(Quick activity in the chat) We recognize that people have different viewpoints on Broader Impacts. When you think about NSF’s Broader Impacts review criterion, what comes to mind? Please add 2-3 words or phrases that come to mind in the chat. [Invite participants to add thoughts in the chat as an icebreaker, then pose next question]
Thank you. Now we can discuss. Considering your experiences as a proposer and PI, what does it mean for research and education to demonstrate broader impacts? You may build on the responses you wrote in the chat or bring up other ideas.
We know NSF has identified a list of examples of Broader Impacts. NSF has also noted the list is meant to be illustrative and PIs and reviewers are not limited to addressing only these examples in their proposals. [Share screen with slide that has the list.] I’ve shared the list of examples and their definitions on the screen. Please take a moment to review the list of examples and give me a thumbs up or nod when you are finished. To what extent do you think these examples capture the meaning of Broader Impacts?
Are any of the examples confusing or hard to interpret?
Are there any aspects of Broader Impacts that you think about differently or are missing from this list?
Experience with Broader Impacts: [40-45 minutes] For the next set of questions, I’d like you to think about your recent NSF proposal submissions.
To what extent did this list of examples from NSF guide how you approached Broader Impacts in your proposals?
How much did you consider or emphasize Broader Impacts examples from NSF’s list when writing your proposals?
How do you think about the relative importance of specific Broader Impacts examples when submitting a proposal to NSF? [stop screen share during this question]
How do you think about balancing the need for considering intellectual merit vs. broader impacts when submitting a proposal?
Before submitting a proposal, have you interacted with the program director about your proposed project, including your approach to Broader Impacts?
IF YES: How did this interaction shape your proposed project?
What feedback (if any) have you received from your reviewers on your proposed approach to Broader Impacts?
Awareness of NSF guidance and trainings related to Broader Impacts: [10-15 minutes] Our final set of questions focus on any guidance you’ve received on Broader Impacts and your participation in any relevant trainings.
(Two polls in the chat) I would like to start with two quick polls. Please add your responses to each question in the chat.
First, are you aware of any guidance NSF has shared related to how to interpret Broader Impacts?
Second, have you participated in any trainings to help you prepare a proposal for NSF that included a focus on Broader Impacts?
Who/where did the guidance or trainings come from? [allow for several illustrative examples]
What was still unclear after reviewing the guidance or participating in the trainings? What areas related to Broader Impacts did you need more guidance on?
Based on your experience, what do you see as some of the factors that might lead to PIs interpreting Broader Impacts differently?
(Poll in the chat) I’d like to pose a final poll. We have discussed how PIs may have varying interpretations of Broader Impacts. How would you characterize the implications of these variations? Would you say these variations advance the merit review process, hinder the process, or some combination of both? Please add your response in the chat.
In what ways do the variations advance the merit review process? [allow for several illustrative examples, if time]
In what ways do the variations hinder the merit review process? [allow for several illustrative examples, if time]
Wrap up: Is there anything else you’d like to discuss or share today related to your understanding and experience with Broader Impacts? [only ask if time remaining]
Thank you: Thank you for your time and openness to sharing your experiences. Your responses will play an important role in our understanding of how NSF interprets and applies the Broader Impacts review criterion across its work. If you would like to share additional feedback on the topics we discussed today, you are welcome to email my colleague Robert Lynn-Green who sent you the calendar invite for this meeting.
Introductions: I would like to start by learning more about each other. Can you each take about 30 seconds to introduce yourself, including your name, where you work, the focus of your work, and how long you’ve been involved with NSF? [call on staff one by one; 6 minutes]
Perception and understanding of Broader Impacts: [20-25 minutes] For the first set of questions, I’d like to understand how you interpret Broader Impacts. According to NSF, “NSF funds scientists and engineers to perform research that advances discovery and innovation. The agency also expects researchers’ work to have broader impacts: the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.”
(Quick activity in the chat) We recognize that people have different viewpoints on Broader Impacts. When you think about NSF’s Broader Impacts review criterion, what comes to mind? Please add 2-3 words or phrases that come to mind in the chat. [Invite participants to add thoughts in the chat as an icebreaker, then pose next question]
Thank you. Now we can discuss. Considering your experiences as a reviewer, what does it mean for research and education to demonstrate broader impacts? You may build on the responses you wrote in the chat or bring up other ideas.
We know NSF has identified a list of examples of Broader Impacts. NSF has also noted the list is meant to be illustrative and PIs and reviewers are not limited to addressing only these examples in their proposals. [Share screen with slide that has the list.] I’ve shared the list of examples and their definitions on the screen. Please take a moment to review the list of examples and give me a thumbs up or nod when you are finished. To what extent do you think these examples capture the meaning of Broader Impacts?
Are any of the examples confusing or hard to interpret?
Are there any aspects of Broader Impacts that you think about differently or are missing from this list?
To what extent does this list of examples from NSF guide your interpretation and assessment of Broader Impacts when reviewing proposals?
How do you think about the relative importance of specific Broader Impacts examples when reviewing a proposal for NSF?
Experience with Reviewing Proposals for NSF: [40-45 minutes] For the next set of questions, I’d like you to think about the most recent proposal review cycles you participated in for NSF.
(Quick activity in the chat) I will start with a quick poll. Please add you response in the chat. Overall, which of these descriptions sounds most like your approach to reviewing recent proposals? A: After reading all or some of the proposal, you have a general sense of whether the project should be funded, and you use the review criteria to justify your review. B: You check the proposal against a list of review criteria before making your summative review. C: Some combination of the two approaches. D: Neither approach.
Can you share more about the guiding principles you use to review proposals? [allow for several illustrative examples, as time allows]
Considering the most recent proposal review cycle you participated in, how did you think about balancing the need for considering intellectual merit vs broader impacts when reviewing proposals? Did your assessments of one inform the other?
Are there specific rubrics, checklists, or other decision aids you use to balance the weight and potential impact for Intellectual Merit vs Broader Impacts? For example, if a project has significant Intellectual Merit, is that alone considered to be a Broader Impact? What about the vice versa?
How do you come to an overall assessment when one criteria is strong and the other weak?
Probe more if needed: Seek to understand their views on how these two criteria are related or different.
I’d like to dig in a bit deeper to understand how you think about Broader Impacts in evaluating proposals.
How do you consider Broader Impacts across proposed projects that are very different in field and scope? For example, how do you compare Broader Impacts for projects focused on different examples of Broader Impacts?
When making an overall assessment of a proposed project’s potential for Broader Impacts, how do you think about the size or value of the potential outcomes versus the probability of those outcomes occurring?
How do you approach examining the applicant’s prior work when coming to your assessment? For example, do you account for their prior demonstration of Broader Impacts, scholarship (such as publications), or other factors?
How do you approach providing PIs written feedback on their proposals? Do you provide detailed feedback on how PIs met the Intellectual Merit and Broader Impacts criteria, and if so, how? If not, why?
Awareness of NSF guidance and trainings related to Broader Impacts: [8-10 minutes] Our final set of questions focus on any guidance you’ve received on Broader Impacts and your participation in any relevant trainings.
(Two polls in the chat) I would like to start with two quick polls. Please add your responses to each question in the chat.
First, are you aware of any guidance NSF has shared related to how to interpret and assess Broader Impacts?
Second, have you participated in any trainings to help you review proposals for NSF that included a focus on Broader Impacts?
Who/where did the guidance or trainings come from? [allow for several illustrative examples]
What was still unclear after reviewing the guidance or participating in the trainings? What areas related to Broader Impacts did you need more guidance on?
Based on your experience, what do you see as some of the factors that might lead to reviewers interpreting and assessing Broader Impacts differently?
(Poll in the chat) I’d like to pose a final poll. We have discussed how reviewers may have varying interpretation of Broader Impacts. How would you characterize the implications of these variations? Would you say these variations advance the merit review process, hinder the process, or some combination of both? Please add your response in the chat.
In what ways do the variations advance the merit review process? [allow for several illustrative examples, if time]
In what ways do the variations hinder the merit review process? [allow for several illustrative examples, if time]
Wrap up: Is there anything else you’d like to discuss or share today related to your understanding and experience with Broader Impacts? [only ask if time remaining]
Thank you: Thank you for your time and openness to sharing your experiences. Your responses will play an important role in our understanding of how NSF interprets and applies the Broader Impacts review criterion across its work. If you would like to share additional feedback on the topics we discussed today, you are welcome to email my colleague Robert Lynn-Green who sent you the calendar invite for this meeting.
Closing (facilitator comments to all participants)
To close, thank the participants for sharing their experiences and perspectives. Thank the moderators and recorders/note takers. End with an opportunity for participants to ask questions of you, time permitting.
Moderator Best Practices (internal guidance/training for focus group facilitators)
The person conducting the focus group will be referred to as the ‘moderator.’ The first skill in moderating is the ability to “initiate and maintain a conversation with a stranger” (Frey & Oishi, 1995). A good moderator uses the following skills:
Be mentally prepared.
Be alert, friendly and free from distractions.
Listen.
Be completely familiar with questions.
Techniques for engaging all participants —
“Let’s take the next four minutes to silently write ideas for this question. [After four minutes] Now, I would like each person to share one idea at a time.”
Control your reactions.
Remain neutral; don’t evaluate or judge in any capacity.
Keep your opinion to yourself.
Never say “that’s good” or “excellent.” Nod your head to encourage dialogue but don’t show agreement with an idea.
Think about what you are communicating verbally and nonverbally.
Keep listening.
Do not defend or justify.
If a participant seems especially emotional (angry, euphoric, etc.), ask the person to describe how they feel.
Validate by saying, “I understand why you would feel that way. Tell me more.”
Validate by saying, “We’re trying to get as much information as possible, so I appreciate you. Would anyone else like to share?”
Offer appropriate questions.
Use pauses and probes.
Ask your question then pause.
Don’t talk to fill the silence — allow people to think about the question.
After someone stops speaking and no one else responds, wait 5 seconds, then call on someone else to comment.
Probes:
“Would you explain further?”
“Tell me more.” or “Would you provide an example?”
“I don’t understand. Tell me more.”
Repeat the question.
Repeat the reply.
Be flexible and consistent.
Moderators balance flexibility in questioning with consistency between and among different focus groups.
If everyone has spoken, ask if there’s anything else, then move on to the next question. Participants should be having a conversation with each other; you are listening to that conversation
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mathematica |
| File Modified | 0000-00-00 |
| File Created | 2024-07-20 |