Broader Impacts Focus Group Interview Protocol

Attachment A ● Data collection instruments
Protocol for Conducting NSF Broader Impacts Focus Groups
Preparation (internal training for focus group facilitators)
When planning and preparing for focus groups, consider the following:
• Ensure that you have a stable internet connection and that you are using a headset for best
audio quality.
• Ensure that you have at least one note taker per group, and that these individuals have the
equipment or supplies they need to record or take notes.
• Moderators should always have their video on. Note takers can choose to have their video
off.
• Be prepared to record the session before you begin the introduction and always ask for
participants’ consent to record before you press record (Note that we intend to ask
permission when recruiting participants but will confirm at the start of each focus group
session).
o If a participant does not consent to being recorded when you ask at the start of the
session, do not record the session. Be sure the note taker knows that there will be
no recording and to take extra care to take detailed and diligent notes.
• Document the following:
o Date and time of the focus group,
o The types of participants included in the focus group (reviewers, accepted PIs,
declined PIs),
o The moderator and note taker, and
o Any participants that are late or no-shows.
• Review the list of “Moderator Best Practices” at the end of this protocol.
Introduction (Focus group facilitators will share at the beginning of each session)
• The moderator introduces her/himself and any others who are playing a working role in the
activities (i.e., note-takers).
• The moderator will provide a high-level background of NSF’s merit review process, including
the project’s purpose.
o About NSF’s merit review: NSF program officers and external experts evaluate
submitted proposals for two main merit review criteria approved by the National
Science Board: (1) Intellectual Merit, referring to the potential to advance
knowledge, and (2) Broader Impacts, referring to the potential to contribute to
society and achieve specific, desired societal outcomes. Of critical importance to
the Foundation is that its merit review process is implemented in a way that is
fair, thorough, competitive, and transparent. However, NSF has found that
principal investigators and reviewers lack clarity about the Broader Impacts
review criterion, despite NSF’s efforts to provide additional guidance.

1

o Project purpose: This project is designed to understand how NSF interprets and
applies the Broader Impacts review criterion across its work. Based on findings
from this project, Mathematica will provide NSF with recommendations on ways
the Foundation could improve the effectiveness of the Broader Impacts review
criterion. This project will produce useful insights to inform ongoing NSF
strategies to meet the goals established in Section 526 of the America Creating
Opportunities to Meaningfully Promote Excellence in Technology, Education,
and Science Reauthorization Act of 2010.
• The moderator will explain the purpose of the event: explain that the purpose of the focus
group is to understand their perspectives and experiences with NSF merit review, especially as it
relates to the Broader Impacts review criterion—which will be used to more clearly understand
specific [Principal Investigator/Reviewer] needs and make recommendations for improvement.
• The moderator will explain confidentiality: explain that we are recording (assuming all have
agreed) and have note takers because we want to make sure to capture their experiences and
ideas. That said, everything that they say is confidential. We ask that they do not share what
others have discussed with people outside of this group.
Ground Rules (Focus group moderators will share at the beginning of each session)
1. First, there are no right or wrong answers. We are interested in your experience and
perspectives on NSF’s merit review and the Broader Impacts review criterion.
2. Second, you do not have to agree with everyone else in this room if that is not how you really
feel. We expect people will have different views on these questions. You don't need to agree
with others, but you must listen respectfully as others share their views.
3. Third, we want you to feel comfortable saying good things as well as critical things. We are
not here to promote a particular way of thinking. We just want to understand your viewpoints.
4. Fourth, we are recording so we ask that you talk one at a time.
6. Confirm with participants that they're comfortable working on a first name basis.
7. Talk to each other.
8. My role as moderator will be to guide the discussion.
******************************************************************************
Questions for PI Focus Group (script to be used for conducting focus group)
1. Icebreaker: (Word Cloud interactive activity). When you think about NSF's Broader
Impacts review criterion, what comes to mind?
2. Introductions: I would like to start by learning more about each other. Can you each take
2 minutes to introduce yourself and provide a little background on your work and
experience with Broader Impacts?
3. Perception and understanding of Broader Impacts: For the first set of questions, I’d like
to understand how you interpret Broader Impacts. According to NSF, "NSF funds
scientists and engineers to perform research that advances discovery and innovation. The
agency also expects researchers' work to have broader impacts: the potential to benefit
society and contribute to the achievement of specific, desired societal outcomes." With
2

this definition of Broader Impacts in mind, NSF has identified some key Broader Impact
outcomes as examples, though PIs and reviewers are not limited to these examples. We
will put this information in the chat and give you 5 minutes to review and reflect before
regrouping. We would like for you to each post your thoughts on the shared Mural Board
if possible or otherwise be willing to discuss your reactions).
Examples of desired outcomes articulated by NSF include (post in chat and have Mural
Board set up for reactions):
Economic competitiveness: Increasing the economic competitiveness of the U.S.
Partnerships: Building partnerships between academia, industry and others.
Infrastructure: Enhancing infrastructure for research and education.
STEM workforce: Developing a more diverse, globally competitive STEM
workforce.
Societal well-being: Improving the well-bein3. g of individuals in society.
Public engagement: Increasing public scientific literacy and public engagement
with STEM.
STEM education: Improving education and educator development — at any level
— in science, technology, engineering and mathematics.
Inclusion: Increasing and including the participation of women, persons with
disabilities and underrepresented minorities in STEM.
a. How do you think about balancing the need for considering intellectual merit vs.
broader impacts when submitting a proposal?
b. How do you think about the relative importance of each Broader Impacts outcome
in submitting a proposal to NSF?
c. Are any of the outcomes listed in the chat confusing or hard to interpret?
d. Are there any aspects of Broader Impacts that you think about differently or are
missing from this list?
4. Experience with Broader Impacts: For the next set of questions, I’d like you to think
about your recent NSF proposal submissions.
a. How much did you consider/emphasize the outcomes listed in the chat when
writing your proposal(s)?
b. In considering our earlier conversation about Broader Impacts that might be
missing from this list; which of those did you emphasize in your proposals and
why?
c. What factors do you think most strongly contributed to funding decisions for your
proposal(s)? Do those factors align with what you heard from your reviewers?
5. Awareness of NSF guidance and trainings related to Broader Impacts: Our final set of
questions focus on any guidance you’ve received on Broader Impacts and your
participation in any relevant trainings.
a. Are you aware of any guidance NSF has shared related to how to interpret
Broader Impacts? Who/where did the guidance come from?
b. Have you participated in any trainings to help you prepare a proposal for NSF that
included a focus on Broader Impacts?
3

c. (If yes to a or b) What was still unclear after reviewing the guidance? What areas
related to Broader Impacts did you need more guidance on?
d. Based on your experience, what do you see as some of the factors that might lead
to PIs interpreting Broader Impacts differently?
e. To what extent do you think that differences in PI interpretations of Broader
Impacts affect the quality of NSF’s merit review process?
f. If NSF were to develop and/or deliver additional guidance on Broader Impacts,
what additional resources would be useful? What communication channels would
you recommend for sharing that information? What kinds of barriers come to
mind about how people may find and access that information?
6. Wrap up: Is there anything else you’d like to discuss or share today related to your
understanding and experience with Broader Impacts?
Questions for Reviewer Focus Group (script to be used for conducting focus group)
1. Icebreaker: (Word Cloud interactive activity). When you think about NSF's Broader
Impacts review criterion, what comes to mind?
2. Introductions: I would like to start by learning more about each other. Can you each take
2 minutes to introduce yourself and provide a little background on your work and
experience with Broader Impacts?
3. Perception and understanding of Broader Impacts: For the first set of questions, I’d like
to understand how you interpret and assess Broader Impacts. According to NSF, "NSF
funds scientists and engineers to perform research that advances discovery and
innovation. The agency also expects researchers' work to have broader impacts: the
potential to benefit society and contribute to the achievement of specific, desired societal
outcomes." With this definition of Broader Impacts in mind, NSF has identified some key
Broader Impact outcomes as examples, though PIs and reviewers are not limited to these
examples. We will put this information in the chat and give you 5 minutes to review and
reflect before regrouping. We would like for you to each post your thoughts on the shared
Mural Board if possible or otherwise be willing to discuss your reactions).
Examples of desired outcomes articulated by NSF include (post in chat and have Mural
Board set up for reactions):
Economic competitiveness: Increasing the economic competitiveness of the U.S.
Partnerships: Building partnerships between academia, industry and others.
Infrastructure: Enhancing infrastructure for research and education.
STEM workforce: Developing a more diverse, globally competitive STEM
workforce.
Societal well-being: Improving the well-bein3. g of individuals in society.
Public engagement: Increasing public scientific literacy and public engagement
with STEM.
STEM education: Improving education and educator development — at any level
— in science, technology, engineering and mathematics.

4

Inclusion: Increasing and including the participation of women, persons with
disabilities and underrepresented minorities in STEM.
a. How do you think about the relative importance of each Broader Impacts outcome
when reviewing a proposal for NSF?
b. Are any of these outcomes confusing or hard to interpret?
c. Are there any aspects of Broader Impacts that you think about differently or are
missing from this list?
4. Experience with Reviewing Proposals for NSF: For the next set of questions, I’d like you
to think about the most recent proposal review cycles you participated in for NSF.
a. How did you think about balancing the need for considering intellectual merit vs
broader impacts when reviewing proposals? Did your assessments of one inform
the other?
i. Are there specific rubrics you use to balance the weight and potential
impact for Intellectual Merit vs Broader Impacts or is it mostly a judgment
call? For example, if a project has significant Intellectual Merit, is that
alone considered to be a Broader Impact? What about the vice versa?
ii. Probe more if needed: Seek to understand their views on how these two
criteria are related or different.
b. How did you specifically consider Broader Impacts across proposed projects that
are very different in field and scope? For example, how do you compare Broader
Impacts for projects focused on national security relative to those more focused
on diversity broader impacts?
c. How do you approach providing PIs written feedback on their proposals? Do you
provide detailed feedback on how PIs met the Intellectual Merit vs Broader
Impacts criteria, and if so, how? If not, why?
d. What factors most strongly contributed to your evaluation of recent NSF
proposals reviewed?
5. Awareness of NSF guidance and trainings related to Broader Impacts: Our final set of
questions focus on any guidance you’ve received on Broader Impacts and your
participation in any relevant trainings.
a. Are you aware of any guidance NSF has shared related to how to interpret and
assess Broader Impacts? Who/where did the guidance come from?
b. Have you participated in any trainings to help you review proposals for NSF that
included a focus on Broader Impacts?
c. (If yes to a or b) What was still unclear to you after reviewing the guidance? What
areas related to Broader Impacts did you need more guidance on?
d. Based on your experience, what do you see as some of the factors that might lead
to reviewers interpreting and assessing Broader Impacts differently?
e. To what extent do you think that differences in reviewer interpretations or
assessments of Broader Impacts affect the quality of NSF’s merit review process?
f. If NSF were to develop and/or deliver additional guidance on Broader Impacts for
reviewers, what additional resources would be useful? What communication

5

channels would you recommend for sharing that information? What kinds of
barriers come to mind about how people may find and access that information?
6. Wrap up: Is there anything else you’d like to discuss or share today related to your
understanding and experience with Broader Impacts?
Closing (facilitator comments to all participants)
To close, thank the participants for sharing their experiences and perspectives. Thank the
moderators and recorders/note takers. End with an opportunity for participants to ask questions
of you, time permitting.
Moderator Best Practices (internal guidance/training for focus group facilitators)
The person conducting the focus group will be referred to as the ‘moderator.’ The first skill in
moderating is the ability to “initiate and maintain a conversation with a stranger” (Frey & Oishi,
1995). A good moderator uses the following skills:
• Be mentally prepared.
o Be alert, friendly and free from distractions.
o Listen.
o Be completely familiar with questions.
• Techniques for engaging all participants —
o “Let’s take the next four minutes to silently write ideas for this question. [After
four minutes] Now, I would like each person to share one idea at a time.”
• Control your reactions.
o Remain neutral; don’t evaluate or judge in any capacity.
o Keep your opinion to yourself.
o Never say “that’s good” or “excellent.” Nod your head to encourage dialogue but
don’t show agreement with an idea.
o Think about what you are communicating verbally and nonverbally.
• Keep listening.
o Do not defend or justify.
o If a participant seems especially emotional (angry, euphoric, etc.), ask the person
to describe how they feel.
o Validate by saying, “I understand why you would feel that way. Tell me more.”
o Validate by saying, “We’re trying to get as much information as possible, so I
appreciate you. Would anyone else like to share?”
• Offer appropriate questions.
o Use pauses and probes.
o Ask your question then pause.
o Don’t talk to fill the silence — allow people to think about the question.
o After someone stops speaking and no one else responds, wait 5 seconds, then call
on someone else to comment.
o Probes:
− “Would you explain further?”
− “Tell me more.” or “Would you provide an example?”
− “I don’t understand. Tell me more.”
6

•

− Repeat the question.
− Repeat the reply.
Be flexible and consistent.
o Moderators balance flexibility in questioning with consistency between and
among different focus groups.
o If everyone has spoken, ask if there’s anything else, then move on to the next
question. Participants should be having a conversation with each other; you are
listening to that conversation

7