Email correspondence

Attachment B ● Data collection communications
See below for email drafts for communications with focus group participants. These email drafts
include recruitment emails, scheduling emails, and thank you emails. Referenced attachments are
presented at the end of this document.
Initial recruitment email
For principal investigators:
Subject: [Response requested] Invitation to Participate in a Focus Group for an NSF Study on
the Broader Impacts review criterion
Attachments: Study Abstract (in pdf form)
Dear [Title] [Last name],
We are excited to invite you to participate in a focus group as a part of an evaluation of the
National Science Foundation’s (NSF) Broader Impacts review criterion. NSF has contracted
with Mathematica to understand evidence-based policies and practices for merit review and
provide NSF with recommendations on ways to improve the effectiveness of the Broader
Impacts criterion. We believe that your expertise and experience as a [Principal Investigator]
who has recently submitted a proposal to NSF will provide a vital perspective to the focus group
and study, which will ultimately strengthen NSF’s merit review process.
Please let us know if this opportunity interests you by replying to this email by [date]. The
opportunity involves participating in a 90-minute virtual focus group hosted on Zoom during the
spring of 2024. We are happy to answer any questions you have and/or you can send questions to
[email protected] if you would like to direct questions to NSF. You can find an abstract of the project
attached to this email. This focus group has been cleared under the Paperwork Reduction Act by
the Office of Management and Budget and the clearance number is 3145-XXXX.
We look forward to hearing from you!
Warmly,
[Mathematica staff name]
[Signature]

For reviewers:
Subject: [Response requested] Invitation to Participate in a Focus Group for an NSF Study on
Broader Impacts
Attachments: Study Abstract (in pdf form)
Dear [Title] [Last name],
We are excited to invite you to participate in a focus group as a part of an evaluation of the
National Science Foundation’s (NSF) Broader Impacts review criterion. NSF has contracted
with Mathematica to understand evidence-based policies and practices for merit review and
provide NSF with recommendations on ways to improve the effectiveness of the Broader
Impacts criterion. We believe that your expertise and experience as a recent reviewer of NSF
proposals will provide a vital perspective to the focus group and study, which will ultimately
strengthen NSF’s merit review process.
Please let us know if this opportunity interests you by replying to this email by [date]. The
opportunity involves participating in a 90-minute virtual focus group hosted on Zoom in the
spring of 2024. We are happy to answer any questions you have and/or you can send questions to
[email protected] if you would like to direct questions to NSF. You can find an abstract of the project
attached to this email. This focus group has been cleared under the Paperwork Reduction Act by
the Office of Management and Budget and the clearance number is 3145-XXXX.
We look forward to hearing from you!
Warmly,
[Mathematica staff name]
[Signature]

Reminder recruitment email
For principal investigators:
Subject: [Reminder] Invitation to Participate in a Focus Group for an NSF Study on Broader
Impacts
Attachments: Study Abstract (in pdf form)
Dear [Title] [Last name],
I hope this email finds you well. I am following up on our invitation to participate in a focus
group as a part of an evaluation of the National Science Foundation’s (NSF) Broader
Impacts review criterion. We believe that your experience as a Principal Investigator who has
recently submitted a proposal to NSF will provide a critical perspective to our study. This focus
group has been cleared under the Paperwork Reduction Act by the Office of Management and
Budget and the clearance number is 3145-XXXX.
Please let us know if you are interested in being a part of our focus group by replying to
this email by [date]. Please do not hesitate to ask any questions you may have.
We are excited to hear from you!
Best regards,
[Mathematica staff name]
[Signature]

For reviewers:
Subject: [Response requested] Invitation to Participate in a Focus Group for an NSF Study on
Broader Impacts
Attachments: Study Abstract (in pdf form)
Dear [Title] [Last name],
I hope this email finds you well. I am following up on our invitation to participate in a focus
group as a part of the National Science Foundation (NSF) evaluation of the Foundation’s
Broader Impacts review criterion. We believe that your experience as recent reviewer of NSF
proposals will provide a critical perspective to our study. This focus group has been cleared
under the Paperwork Reduction Act by the Office of Management and Budget and the clearance
number is 3145-XXXX.
Please let us know if you are interested in being a part of our focus group by replying to
this email by [date]. Please do not hesitate to ask any questions you may have.
We look forward to hearing from you!
Warmly,
[Mathematica staff name]
[Signature]

Follow up recruitment email
Subject: Thank You for Accepting Invitation to Participate in Focus Group for NSF Study on
Broader Impacts
Attachments: Focus group discussion topics (in PDF form)
Dear [Title] [Last name],
Thank you for accepting our invitation to participate in a focus group as a part of an
evaluation of the National Science Foundation (NSF’s) Broader Impacts review criterion.
By participating in the focus group, you will have a chance to share your experiences and help
contribute to strengthen NSF’s merit review process. This focus group has been cleared under
the Paperwork Reduction Act by the Office of Management and Budget and the clearance
number is 3145-XXXX.
We would like to note that we are planning to record the session for note taking, but these videos
will be kept entirely confidential. If this affects your ability or willingness to participate, please
let us know.
We will send out scheduling information [insert timeline]. Please do not hesitate to ask any
questions you may have. We are looking forward to working with you soon!
Kind regards,
[Mathematica staff name]
[Signature]

Initial scheduling email
Subject: [Response Requested] Scheduling Focus Group for NSF Study on Broader Impacts
Attachments: None
Dear [Title] [Last name],
Thank you again for accepting our invitation to participate in a focus group as a part of an
evaluation of the National Science Foundation’s (NSF) Broader Impacts review criterion.
We are excited to have you participate and share your experience with NSF’s merit review
process. We are in the process of scheduling the virtual focus group and have narrowed down the
date and time to three potential time slots and have included a survey linked in the “other”
category if your availability does not align with the three proposed times. The three time slots
and the “other” category are listed below.
1)
2)
3)
4)

[Day of week, date, time]
[Day of week, date, time]
[Day of week, date, time]
Other [Insert survey asking for general availability]

We are convening participants across multiple time zones, so we ask that you respond to this
email confirming your availability for each of the three time slots (or fill out the survey
linked in the “other” category if your availability does not align) by [date]. Once we
determine the date and time with the most availability from the group, we will send a calendar
invitation and Zoom link for the date selected.
We would like to reiterate that we are planning to record the session for note taking, but these
videos will be kept entirely confidential. If this affects your ability or willingness to participate,
please let us know. This focus group has been cleared under the Paperwork Reduction Act by the
Office of Management and Budget and the clearance number is 3145-XXXX.
As always, please do not hesitate to ask any questions you may have about this process. We
greatly appreciate your time and participation.
Kind regards,
[Mathematica staff name]
[Signature]

Reminder scheduling email
Subject: [Reminder] Scheduling Focus Group for NSF Study on Broader Impacts
Attachments: None
Dear [Title] [Last name],
I hope this email finds you well. We are excited to have you participate in a focus group as a part
of an evaluation of the National Science Foundation’s (NSF) Broader Impacts review criterion.
This is a reminder to please confirm your availability for each of the three time slots (or fill
out the survey linked in the “other” category) below by [date]. Once we determine the date
and time with the most availability from the group, we will send a calendar invitation and Zoom
link for the date selected. This focus group has been cleared under the Paperwork Reduction Act
by the Office of Management and Budget and the clearance number is 3145-XXXX.
1)
2)
3)
4)

[Day of week, date, time]
[Day of week, date, time]
[Day of week, date, time]
Other [Insert survey asking for general availability]

We greatly appreciate your time and participation.
Kind regards,
[Mathematica staff name]
[Signature]

Follow up scheduling email
Subject: Focus Group for NSF study on Broader Impacts Scheduled
Attachments: Focus group discussion topics (in PDF form)
Dear [Title] [Last name],
Thank you again for accepting our invitation to participate in a focus group as a part of the
National Science Foundation (NSF) evaluation of the Foundation’s Broader Impacts review
criterion.
The focus group has been scheduled for:
[Day of week, date, time]
We would like to reiterate that we are planning to record the session for note taking, but that
these videos will be kept entirely confidential. If this affects your ability or willingness to
participate, please let us know. This focus group has been cleared under the Paperwork
Reduction Act by the Office of Management and Budget and the clearance number is 3145XXXX.
Shortly, you will receive a calendar invitation for the date of the focus group with a Zoom link.
Please accept the invitation. When it is time to join the meeting, please click the Zoom link
attached to the invitation. We ask that everyone aim to join on time so that we have ample time
to conduct the focus group. We are excited to have you join us and contribute to this very
important study!
Kind regards,
[Mathematica staff name]
[Signature]

Thank you email
Subject: Thank You for your Participation in our [Interview/Focus Group] for NSF Study on
Broader Impacts
Attachments: None
Dear [Title] [Last name],
Thank you so much for participating in a [focus group/interview] as a part of an evaluation
of the National Science Foundation’s (NSF) Broader Impacts review criterion. We are very
appreciative of your time and willingness to share your perspective on Broader Impacts. Findings
from the [interviews/focus groups] will help to inform recommendations to NSF regarding the
Broader Impacts criterion. Please do not hesitate to reach out with any questions or additional
input you may have.
Warm regards,
[Mathematica staff name]
[Signature]
Email attachments (as specified above and to be attached as a PDF)
Study Abstract
This work, mandated by Congress, is designed to understand evidence-based policies and
practices for merit review and assess the alignment of the National Science Foundation’s (NSF)
merit review process with these policies and practices. NSF has contracted with Mathematica to
conduct a literature review to summarize evidence-based merit review policies and practices
(including, but not limited to, Broader Impacts) and a process evaluation to understand how the
Foundation interprets and applies the Broader Impacts review criterion across its work. Based on
findings from these activities, Mathematica will provide NSF with recommendations on ways the
Foundation could improve the effectiveness of the Broader Impacts review criterion. The process
evaluation will comprise three main components: (1) interviews and focus groups with NSF
staff, National Science Board members, principal investigators, reviewers, and early-career
scholars; (2) a document review; and (3) analyses of extant data. Mathematica will also convene
a technical working group of external experts to provide guidance and feedback on the
evaluation. This project will produce useful insights to inform ongoing NSF strategies to meet
the goals established in Section 526 of the America Creating Opportunities to Meaningfully
Promote Excellence in Technology, Education, and Science Reauthorization Act of 2010.
Focus Group Discussion Topics
Planned discussion topics include:
1) How you understand and interpret NSF’s Broader Impacts review criterion.
2) How you applied or assessed the Broader Impacts review criterion in your most recent
proposals or reviews.
3) What guidance, training, or feedback related to the Broader Impacts review criterion (a)
you have received from NSF and (b) would be helpful to have.