SNOFO-Pilot_Evaluation Plan_0313

SimplerNOFO Pilot Evaluation Plan

Last updated March 5, 2024

Objectives:

• Primary Objective: Simplify and reorganize notice of funding opportunity (NOFO) content to help qualified applicants, especially those representing underserved communities, more easily access, read, and understand NOFO objectives, program and application requirements, and instructions.

• Secondary objective: Standardize applicable NOFO content and provide NOFO writing training and coaching so grant makers can create clear and concise NOFOs more efficiently and receive strong submissions from qualified applicants, especially from applicants representing underserved communities.

Metrics:

In preparation for evaluating the overall efforts and outcomes of the SimplerNOFO Pilot, we identified various metrics and classified them under two categories: SimplerNOFO Pilot Outcomes and SimplerNOFO Pilot Process.

For each metric, we specified the timing, collection method(s), and data owners to set the stage for measuring them.

The tables below outline the metrics and the pertinent details, including the data sources for both SimplerNOFO Pilot and historical data (where applicable).

Note. The months in the “Timing” column can be adjusted based on the final publish dates of the SimplerNOFO Pilot.

Section 1: SimplerNOFO (SNOFO) Pilot Outcomes

Link to applicant survey: Notice of Funding Opportunity (NOFO) Feedback Survey

Metrics	Timing	Collection Method	Data Collection Owner	Applicant Survey Question
Objective: SNOFOs are accessible and usable
Rating of overall SNOFO quality Baseline: overall standard NOFO customer rating score	Five business days after application is submitted (Start of Mar 2024)	Applicant Survey	Evaluation Team	3. How would you rate the overall quality of the NOFO you applied for on a scale of 1 to 10?
Amount of time spent applying for funding on SNOFOs Baseline: Amount of time spent applying for funding on standard NOFOs	Five business days after application is submitted (Start of Mar 2024)	Applicant Survey	Evaluation Team	14. Approximately how many hours did it take you to complete your application, including the time spent preparing attachments?
Average reading time for SNOFOs Baseline: Average reading time for historical NOFOs	Immediately after publication	SimplerNOFO Pilot Tracker	Evaluation Team
Level of effort spent completing the SNOFO application Baseline: Level of effort spent completing the standard NOFO application	Five business days after application is submitted (Start of Mar 2024)	Applicant Survey	Evaluation Team	13. How would you describe your overall level of effort spent to complete the application?
Ease of SNOFO navigation rating Baseline: Ease of standard NOFO navigation rating	Five business days after application is submitted (Start of Mar 2024)	Applicant Survey & interviews	Evaluation Team	9. How clear were each of the NOFO features? (Ease of navigation)
Assistance required to apply for SNOFOs Baseline: Assistance required to apply for standard NOFOs	Five business days after application is submitted (Start of Mar 2024)	Applicant Survey	Evaluation Team	12. Did you hire someone, such as a grant writer, to help you apply to this NOFO?
Total number of applications for SNOFOs Baseline: Total number of applications for historical NOFOs	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Awarding Agency Program Officers	n/a
# and % of applicants who are new to OpDiv Baseline: # and % of applicants who are new to OpDiv from historical NOFO	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Awarding Agency Program Officers	n/a
# and % of awards going to applicants who are new to OpDiv Baseline: # and % of awards going to applicants who are new to OpDiv for historical NOFO	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Grant-Solutions/ Grants.gov and Awarding Agency Program Officers	n/a
Objective: Applications from applicants representing underserved communities are strong
# and % of applications from a diverse range of applicants, including small organizations, minority-owned businesses, and underserved communities	Five business days after the applicant’s submission of their application (Start of Mar 2024)	Applicant Survey	Evaluation Team	20. Does your organization represent or support underserved communities? (Defined according to EO13985)

Objective: SNOFOs are readable and comprehensible
SNOFO usability rating by NOFO components Baseline: Standard NOFO usability rating by NOFO components	Five business days after the applicant’s submission of their application (Start of Mar 2024)	Applicant Survey	Evaluation Team	7. How easy was it to understand the NOFO components?
SNOFO Flesch Kincaid grade level reading score Baseline: historical NOFO Flesch Kincaid grade level reading score	Immediately after publication	SimplerNOFO Pilot Tracker	Evaluation Team
SNOFO word count Baseline: historical NOFO word count	Immediately after publication	SimplerNOFO Pilot Tracker	Evaluation Team
SNOFO page length Baseline: historical NOFO page length	Immediately after publication	SimplerNOFO Pilot Tracker	Evaluation Team
SNOFO words per sentence Baseline: historical NOFO words per sentence	Immediately after publication	SimplerNOFO Pilot Tracker	Evaluation Team
SNOFO passive voice count Baseline: historical NOFO passive voice count	Immediately after publication	SimplerNOFO Pilot Tracker	Evaluation Team
Types of questions and comments received by program points-of-contact and help desk	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Evaluation Team
Objective: Applicants are qualified and applications are strong
# and % of applicants that meet organization eligibility criteria Baseline: # and % of applicants that meet organization eligibility criteria compared to historical NOFO	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Awarding Agency Program Officers
# and % of applicants that meet project eligibility criteria Baseline: # and % of applicants that meet project eligibility criteria compared to historical NOFO	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Awarding Agency Program Officers
# and % of SNOFO applications that met the basic minimum requirements Baseline: # and % of historical NOFO applications that met the basic minimum requirements	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Awarding Agency Program Officers
SNOFO merit review score medians Baseline: Historical NOFO merit review score medians	Within two weeks of NOFO award date (Start of Jun 2024)	Program Officer Form	Awarding Agency Program Officers

Section 2: SimplerNOFO Pilot Development Process

Metrics	Timing	Collection Method	Data Collection Owner	Survey Question
Objective: Grant makers can write clear and concise NOFOs using new tools, trainings, and best practices
Writer ratings of overall SNOFO pilot experience	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	How would you rate your overall experience producing a simplified NOFO? [Very satisfied, satisfied, neither satisfied nor unsatisfied, unsatisfied, very unsatisfied]
# and % of SNOFO writers who attended “Science of Writing NOFOs for Busy Applicants” training	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	Did you attend the “The Science of Writing NOFOs for Busy Applicants” training led by Todd Rogers? [Yes – go to next question; No – skip next question]
SNOFO writer rating of “Science of Writing NOFOs for Busy Applicants” training	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	How helpful was the “The Science of Writing NOFOs for Busy Applicants” training led by Todd Rogers? [Very helpful, helpful, neither helpful nor unhelpful, unhelpful, very unhelpful]
# and % of SNOFO writers who attended “Plain and Simple” training	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	Did you attend the training “Plain and Simple” training led by Camille Cain? [Yes – go next question; No – skip next question]
SNOFO writer rating of “Plain and Simple” training	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	How helpful was the “Plain and Simple” training led by Camille Cain? [Very helpful, helpful, neither helpful nor unhelpful, unhelpful, very unhelpful]
# and % of SNOFO writers who used resources available on the website	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	Did you utilize the SimplerNOFOs information available on Simplergrants.hhs.gov? [Yes – go to next question; No – skip next question]
SNOFO writer rating of website resources	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	How helpful was the information available on Simplergrants.hhs.gov? [Very helpful, helpful, neither helpful nor unhelpful, unhelpful, very unhelpful]
SNOFO writer rating of meeting training and resource objectives	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	The SimplerNOFOs training and resources helped me: Learn plain language principles and writing tips. [Agree, Disagree] Understand the guiding principles for scoping Simpler NOFOs. [Agree, Disagree] Apply plain language to NOFOs. [Agree, Disagree] Produce a simpler NOFO that I’m proud of. [Agree, Disagree]
SNOFO writer ratings of design of their SNOFO	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	Rate your overall satisfaction with the design(s) of your SimplerNOFO(s). [Very satisfied, satisfied, neither satisfied nor unsatisfied, unsatisfied, very unsatisfied]
SNOFO writer ratings of communication from SimplerNOFOs team	Within 5 – 7 days of the NOFO post date	Writer Survey	Evaluation Team	How satisfied are you with the communication you received from the SimplerNOFOs team throughout the process? [Very satisfied, satisfied, neither satisfied nor unsatisfied, unsatisfied, very unsatisfied]
Average length of time to produce NOFOs from start of edits to production and publication	Two weeks after publication of NOFO to ensure amendments are reflected (Start of Jan 2024)	Simpler NOFO Pilot Tracker	Evaluation Team	n/a
Time spent in draft	Two weeks after publication of NOFO to ensure amendments are reflected (Start of Jan 2024)	Simpler NOFO Pilot Tracker	Evaluation Team	n/a
Time spent in approval	Two weeks after publication of NOFO to ensure amendments are reflected (Start of Jan 2024)	Simpler NOFO Pilot Tracker	Evaluation Team	n/a
SNOFO Coach ratings of writing teams’ ability to use content guides	CDC: March HRSA: Early April ACF: Late April Additional OpDivs: May/June	Coach Retros	Yes (Simpler NOFO Pilot)	How well did the writers in the OpDivs follow a content guide? [Very well, well, neither well nor unwell, unwell, very unwell]
SNOFO Coach ratings of writing teams’ ability to follow plain language best practices	CDC: March HRSA: Early April ACF: Late April Additional OpDivs: May/June	Coach Retros	Yes (Simpler NOFO Pilot)	How well did the writing teams follow plain language best practices when creating NOFO content? [Very well, well, neither well nor unwell, unwell, very unwell]
SNOFO Coach ratings of writing teams’ ability to articulate program goals	CDC: March HRSA: Early April ACF: Late April Additional OpDivs: May/June	Coach Retros	Yes (Simpler NOFO Pilot)	How well did writing teams articulate the goals of their program? [Very well, well, neither well nor unwell, unwell, very unwell]
SNOFO Coach ratings of content guides	CDC: March HRSA: Early April ACF: Late April Additional OpDivs: May/June	Coach Retros	Yes (Simpler NOFO Pilot)	How well did the content guides do their job? [Very well, well, neither well nor unwell, unwell, very unwell]

5