Download:
pdf |
pdfPUBLIC COMMENTS ON
NSF MAJOR FACILITIES GUIDE (MFG) DRAFT, DECEMBER 2018
& NSF FINANCIAL DATA COLLECTION TOOL
WITH NSF RESPONSES
Each comment is addressed with one of the following NSF Responses
•
•
•
Addressed in this version with an explanation of how the comment was addressed
No change with an explanation of why
Will be assessed and addressed in later version
Cover Page & Legend
Page 1 of 37
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
1.
Sec Para,
page no.
General
Source
AUINRAO
(Beasley)
2.
General
AUINRAO
3.
General
Woods
Hole-OOI
MFG Language/Observation
Comment
NSF Response/Resolution
Who is the audience for this document now? If it is
meant to be project proponents – ok, it does a lot of
that, but also has long sections on the way NSF is
governing MREFC, how NSF is doing its internal
processes, etc. It is good knowing that stuff, but it
increases the bulk of the document and therefore
the challenges to efficiently extract information
from it.
This document may be too long now … there’s a lot
of great stuff, but it’s not easy to read or navigate.
There are long detailed sections in there (like Risk
management) that probably would be better off in
their own documents, and merely summarized
here? The LFM used to be about policy and
summaries of processes, now it seems to be leaning
towards recording the processes, which is making it
unwieldy?
Preference: real page numbers vs section-specific
numbers (which can get deep and long)
No change.
The audience is both internal and external.
The Guide is intended to summarize internal
workings at a high level, so Recipients
understand the process. NSF has several
internal process documents that the
community does not see.
There are multiple instances of "Large Facility,"
"LFOs, "HLFO," and "Large Facilities" throughout the
document, including the header, 2.1.6-15 through 18. 2.1.6, 4.6.2-2, 2.1.4-5, 2.1.6-1 through -3, and
page 8-2. I believe the intention was to change
these instances to align with new terminology
"Major Facility."
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
NSF has equal concerns about it growing in
size and intend to pare down some sections
as we enhance others. We may bring in a
professional editor as part of a future
revision to help with the organization and
areas of duplication.
No change.
With the size of the Guide and alignment of
content type to Sections and Sub-sections,
we find the use of the section number in the
page number to be more helpful than
confusing.
Accepted.
Yes, the intention was to change “Large
Facility” to “Major Facility” throughout the
document except the organizational name
of the Large Facilities Office (LFO) has not
changed and remains in the Guide as LFO
and Head of the LFO (HLFO). A word search
was conducted, and associated edits were
made.
Page 2
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
4.
Sec Para,
page no.
General
Source
Trusted
CI
MFG Language/Observation
Comment
NSF Response/Resolution
The purpose of our detailed comments and
suggested changes or additions is to aid in usability
and readability, as well as alignment with Trusted
CI’s guidance to the community.
No change.
“Information Security” and “Cybersecurity”
are defined as equivalent terms in Footnote
3 on page 6.3.2-1 which states “For the
purposes of this section, there is no
distinction among the terms “information
security,” “cybersecurity,” and “IT security”
as referenced in the award terms and
conditions. However, this section specifically
addresses digital information.”
Suggested change: Replace “information security”
with “cybersecurity” throughout or define them as
being equivalent terms.
Discussion: Cybersecurity and information security both used but not explicitly described as equivalent.
5.
General
Trusted
CI
Justification: Clarity and consistency
Throughout the document: Suggested change: Add
page numbers to the document
Discussion: The lack of page numbers makes
referencing or communicating about the text in the
document more difficult.
6.
General
BOAC
Subcom
mittee
7.
General
NSF
BFA-DGA
Justification: Improve ease of communication about
parts of the text.
NSF initiate a dialogue with the recipient community
in preparation for the establishment of a set of core
competency recommendations for recipient staff
who support the administrative and management
aspects of large facilities projects.
“Grant/Agreement Officer” should be “Grants and
Agreements Officer”
“Contract Officer” & “Contracts Officer” should be
“Contracting Officer”
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
There are page numbers. It is the subsection number and then consecutive page
number. For example, pages 6.3.3-1 and
6.3.3-2 discuss Governance under Section
6.3 Guidelines for Cyber-Security of NSF’s
Major Facilities.
Accepted and will be addressed in later
version.
This will be codified as part of an interim
update to the MFG by adding a new section
on "Key Personnel".
Accepted.
Changed throughout the document.
Page 3
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
General
NSF
BFA-DGA
9.
1.1,
1.1-1
AUINRAO
10.
1.1,
1.1-1
AUINRAO
11.
1.1,
1.1-2
AUINRAO
12.
1.1,
1.1-2
NSF
BFA-DGA
8.
MFG Language/Observation
“NSF makes awards to external
Recipients that include nonprofit
organizations and universities to
undertake construction,
management, and operation of
facilities. Such awards frequently
take the form of cooperative
agreements but may also be made
in the form of contracts. The
reasons underlying the selection
of the cooperative agreement as
the appropriate funding
mechanism are: …”
“NSF makes awards to external
Recipients that include nonprofit
organizations and universities to
undertake construction,
management, and operation of
facilities.”
“The cooperate agreements also
affords flexibility to tailor projectspecific requirements and
performance metrics.”
“The cooperate agreements also
affords ...”
Comment
NSF Response/Resolution
References to “Federal grant” should be “Federal
award” and the “the applicable award terms and
conditions”;
“assistance agreement” should be “assistance
award”;
“government funds” should be “Federal funds”;
“cooperative service agreement” should be
“cooperative support agreement”
The Guide says cooperative agreements are “the
appropriate” approach, which seems to imply
contracts aren’t … but you admit you do have
contracts. “an appropriate funding mechanism”
better?
Accepted.
MFG talks about NSF using CAs with a range of
organizations, but you don’t list “for profit” orgs…
omission? (polar: Lockheed martin)
Accepted.
Awards can be made to “for profit”
organizations as indicated later in the Guide.
Sentence was edited to include “private
sector”.
“affords” should be “afford”
Accepted.
Removed “The” and changed “affords” to
“afford”.
“cooperate agreements” should be “cooperative
agreement”
Accepted.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
The language was revised for clarification cooperative agreements are the preferred
award instrument. The reasons for selection
of cooperative agreement are given in the
bullets on page 1.1-1. If these conditions are
not present, then a contract may be
appropriate.
Page 4
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
1.1,
1.1-2
AUINRAO
Seems like the guide talks about more than just
“planning and management”... initiation, planning,
evaluation, execution, performance monitoring,
lifecycle, etc.
14.
1.1,
1.1-2
AUINRAO
15.
1.1,
1.1-3
AUINRAO
Accepted.
Added the words “through their full life
cycle” after “major facilities” to indicate
from initiation through execution to the end
of the life cycle (divestment).
Accepted.
A footnote was added listing each life cycle
stage and referencing Section 2 of the Guide
for a description of each stage.
No change.
A footnote was added to define life cycle.
16.
1.1,
1.1-3
AUINRAO
17.
1.1,
1.1-3
AUINRAO
“The Major Facilities Guide (MFG)
contains NSF policy on the
planning and management of
major facilities. The purpose of the
Guide is to: …”
“…required policies and
procedures as well as pertinent
guidance and practices at each
stage of a facility’s life cycle.”
“The R&RA account is used to
support other activities involving a
major facility that the MREFC
Account cannot support, including
planning and development,
design, operations and
maintenance, and scientific
research.”
“Facilities and infrastructure
projects constructed or acquired
with funds from the R&RA (and/or
leveraged with EHR Accounts) with
a total project cost (TPC) greater
than $100 million or that require
National Science Board (NSB)
authorization; whichever is less.”
“Facilities and infrastructure
projects constructed or acquired
with funds from the R&RA (and/or
leveraged with EHR Accounts) with
a total project cost (TPC) greater
than $100 million or that require
National Science Board (NSB)
authorization; whichever is less.”
13.
Sentence uses the phrase “life cycle”, but not clearly
defined
This sentence kind of defines life cycle here… move
up?
Refers to the National Science Board – is that body
defined somewhere? (overall question – are the
committees/boards/departments/individuals
involved all named somewhere?
No change.
Section 2.1.6 describes the organizations
and individuals with responsibilities
associated with oversight of NSF Major
Facilities.
$100M before some external constraints added?
Seems a bit high to me. And there are $25M
projects with horrible externalities (international
partnerships) where some monitoring might be
relevant. I know this won’t be a popular opinion
here, but defining the threshold purely by cost, and
setting that number so high, seems… brave.
Conversely, too much process on small projects is
annoying and wasteful.
Accepted.
A reference to Section 5 Guidance for MidScale Research Infrastructure Project of the
Guide was added to this section. The
policies in this Guide are tailored for midscale projects. Section 1.4.4 defined midscale projects.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 5
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
1.1,
1.1-3
AUINRAO
LFO acronym used … defined?
Accepted.
Revised to “Large Facilities Office (LFO)
Liaison”
19.
1.2,
1.2-1
AUINRAO
hierarchy of documentation … wouldn’t there be
some higher docs? NSF organic act, NSF
mission/charter, NSF strategic plan, ??
20.
1.2,
1.2-1
AUINRAO
“If, on a case-by-case basis,
departures from the policies in
this Guide are considered
necessary or prudent, the
Recipient must provide a written
justification and discuss proposed
deviations with the Program
Officer, LFO Liaison, and
Grants/Agreements Officer or
Contracting Officer as early as
possible.”
“The MFG requirements flow from
other NSF policies and statutory
requirements. The hierarchy of
documentation, in order of
precedence, is as follows: …”
“All facilities projects require merit
review, programmatic/technical
review, and a substantial approval
process. This level of review and
approval differs substantially from
standard grants, as does the level
of oversight needed to ensure
appropriate and proper
accountability for federal funds.
The policies, requirements,
recommended procedures, and
best practices presented herein
apply to any facility large enough
to require interaction with the
NSB or any facility so designated
by the Director, the Deputy
Director, or the Assistant
Director/Office Head of the
Originating Organization(s).3
No change.
Impact of NSF’s Organic Act on major
facilities is given in Section 1.1. The MFG
has no requirements that flow down from
this Act.
No change.
Footnote 3 states “See Section 2.1.6 for
definition of this and other key terms. …
Readers not familiar with NSF and its
processes should review this material before
proceeding.”
18.
“merit review, programmatic/technical review,
substantial … all jargon?”
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 6
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
1.2,
1.2-2
AUINRAO
Asks POs for feedback, don’t tell them how to
deliver it … “please direct your comments to XXX”
Accepted.
Added a reference to NSF Knowledge
Management program which identifies the
process for updating policies and
procedures, such as this Guide.
22.
1.3,
1.3-1
AUINRAO
Para 1 third bullet, comma missing after facility
Accepted.
23.
1.4.1,
1.4-1
AUINRAO
Why is “Major Facilities” both capitalized?
24.
1.4.2,
1.4-1
AUINRAO
Accepted.
Associated edits were made throughout the
guide.
No change.
These references help maintain the
historical record for NSF.
25.
1.4.3,
1.4-1
AUINRAO
“Program Officers (PO) are
encouraged and expected to
continue to identify and adopt
best practices aimed at improving
NSF oversight and Recipient
management of major facilities
projects …”
“Section 3 describes the
requirements for preparing and
following the various detailed
management plans required
during the life cycle of a major
facility including Recipient’s plans
and guidance for NSF’s Internal
Management Plans (IMPs).”
“Major Facilities and mid-scale
projects are subsets of research
infrastructure.”
“NSF Director Memo NSB-2016-46
to the National Science Board
dated October 20, 2016 informed
the Board of the Director’s
decision to reduce the Total
Project Cost (TPC) threshold for
MREFC account eligibility to $70
million. This modification to the
previous threshold of 10% of a
Directorate’s or Office’s Current
Plan was intended to enable
innovative infrastructure
projects.”
“Per Section 110 of the 2017
American Innovation and
Competitiveness Act (AICA), a
major multi-user research facility
…”
21.
Remove history lesson, just state current limits at
time of document freeze?
Remove history lesson, just state current limits at
time of document freeze?
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
These references help maintain the
historical record for NSF.
Page 7
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
1.4.3 &
1.4.4,
1.4-1
AUINRAO
Guide makes this distinction about “multi-user”. Is
that real? generally big facilities are for multiple
science goals, but so are some of the midscale
facilities (in fact, they go out of their way to find
additional uses for their data). the “multi-user”
definition is a correlation, not causal?
Accepted.
A clarification has been added. Multi-user is
a congressional term. The Guide
intentionally shortens to simply “major
facility”.
27.
1.4.5,
1.4-2
AUINRAO
Recomp thing non sequiter ? And is an award issue,
not a facilities issue. this text seemed lost here.
No change.
This language is included in this Section
Applicable Legislation and NSF Policy
because it is policy set by the NSB regarding
competition, renewal, and divestment of
major facilities.
28.
2.1.2,
2.1.2-1
AUINRAO
“Per Section 109 of AICA, a midscale project means research
instrumentation, equipment, and
upgrades to major research
facilities or other research
infrastructure investments that
exceeds the maximum funded by
the Major Research
Instrumentation program (MRI)
and are below that of a major
multi-user research facility
project.”
“NSB statement 2015-45 and
resolution 2015-46 address
recompetition of major facilities.
The NSB issued a statement that
the question of whether to
recompete or not should be
assessed at the time of every
potential renewal.”
Ahem.. 7 character page number ☹)
29.
2.1.2,
2.1.2-1
AUINRAO
No change.
With the size of the Guide and alignment of
content type to Sections and Sub-sections,
we find the use of the section number in the
page number to be more helpful than
confusing.
No change.
Section 2.1.1 defines MREFC funding is
specifically for the construction of major
science and engineering infrastructure. The
use of “Project” refers to the Construction
Stage which has a defined start and end.
26.
“To be eligible for consideration
for MREFC funding, each
candidate project should
represent …”
“In addition, a candidate project
should…”
In para 1 and 2, it talks about projects, do you mean
facilities?
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 8
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
2.1.3,
2.1.3-2
NSF
BFA-LFO
31.
2.1.3,
2.1.3-3
NSF
BFA-LFO
32.
2.1.3,
2.1.3-4
AUINRAO
33.
2.1.3,
2.1.3-5
AUINRAO
34.
2.1.4,
2.1.4-1
AUINRAO
35.
2.1.5,
2.1.5-1
AUINRAO
30.
MFG Language/Observation
“The Concept of Operations Plan
(including robust operations and
maintenance cost estimates and
agreements between parties for
funding, data sharing, etc.) should
be in place in preparation for
entering this stage.
“It is recommended that the
Sponsoring Directorate develop a
plan that follows NSF policy on
divestment decisions…”
“Following successful research and
development by scientists and
engineers in an educational
institution, the entire project may
then be further designed and
constructed by an award made
directly to a competent managing
organization, including industry.”
This section, to be written, will
illustrate when various
preconstruction planning activities
…”
Comment
NSF Response/Resolution
The process to exit from Development Stage to
transition to Design Stage is a recommendation
from the CORF to the Director based on a strategic
review and NSF Director approval.
For projects with other funding for development
and design, can enter at CDR or PDR based on
technical readiness and strategic review by the
CORF.
In para 2 – At the Operations Stage, it talks about
the Concept of Operations doc being in place –
suggest at this point, detailed operation plans and a
full suite of maintenance / development plans
should be in place.
Accepted.
“Sponsoring Directorate” defined?
Accepted.
The guide used “Sponsoring Directorate”
and “Originating Organization”
interchangeable. The guide has been
revised to use “Sponsoring Organization”
which is defined in Section 2.1.6.1.
No change.
Awards can be made the private sector.
Section 1.1 was revised in response to
comment above.
Here you seem to suggest for-profits can be
involved. (see comment above).
TBD
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
Accepted.
Agree the that Concept of Operation Plan
that is initiated during the Design Stage
should be finalized prior to start of
Operations. The language has been revised
to clarify.
No change.
Comment is an observation. This Section is
planned to be addressed in a future version
of this Guide.
Page 9
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
Comment
NSF Response/Resolution
2.1.6
AUINRAO
(Beasley)
The section of how LFO, CORF, senior NSF layers
interact with projects – was somewhat difficult to
understand. The landscape IS difficult to
understand, I know. I wonder if this material would
be better as highly summarized here, more detail in
some supporting doc?
Will be assessed and addressed in later
version.
We may bring in a professional editor as part
of a future revision to help with the
organization and areas of duplication.
37.
2.1.6,
2.1.6-1
through
2.1.6-22
NSF
BFA-DGA
DGA is not involved with major facility awards. CSB
is responsible for cooperative agreement awards for
the Major Facilities. Recommend references to DGA
be removed from the MFG.
38.
2.1.6.1,
2.1.6-4
AUINRAO
(Beasley)
Accepted.
Section 5 of the MFG provides guidance for
mid-scale projects which are funded through
both DGA & CSB. References to DGA were
retained but clarifications were added.
Titles of figures were revised to include
“mid-scale research infrastructure”. DGA
was removed from Figure 2.1.6-4 since IPTs
are not associated with mid-scale projects.
On page 2.1.6-12, changes to “The G/AO is
administratively part of DACS in BFA, except
for mid-scale projects where they may be
part of DGA (See Section 5 of this Guide).”
No change.
Footnote 3 on page 1.2-1 references this
Section for descriptions of the NSF
organizations and officers.
39.
2.1.6.1,
2.1.6-4
AUINRAO
40.
2.1.6.1,
2.1.6-8
NSF
BFADACS
36.
MFG Language/Observation
“NSB – Establishes policy, reviews
and authorizes MREFC Account
budgets, and reviews and
authorized specific MREFC
projects for funding.”
“Figure 2.1.6-3 NSF organization
chart showing policy and approval
bodies for major facilities.”
Here those definitions of stakeholders… could
reference earlier … (NSB defined?)
All these panels and boards are confusing. It looks
like you’ve got multiple threads of governance and
responsibility, and I appreciate the effort to map it
out in some non-conflicting way, but….
(understand, this may be where we are at present,
but I’m pretty sure if I provided you a project org
chart like this, you’d want it thinned/trimmed).
Table 2.1.6-1: Change “Cost Proposal Review
Document (CPRD)” to “Decision Memo” to align
with internal terminology.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
Figures 2.1.4-1 and 2.1.4-2 provide a
mapping of the Panels and Boards to the
major facility life cycle stages and NSF
oversight responsibilities. A footnote was
added here to refer to these figures.
Accepted.
Page 10
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
41.
Sec Para,
page no.
2.1.6.2,
2.1.6-15
Source
MFG Language/Observation
Comment
NSF Response/Resolution
AUINRAO
“The CORF advises the NSF
Director on all aspects of NSF
major and mid-scale facilities
throughout their life-cycle and
collaborates with all at NSF who
are involved in oversight and
assistance for the NSF research
facilities portfolio.”
“The LFO has the institutional
authority and resources to
effectively develop mandatory
policies, practices and procedures,
which are approved by senior
management, for all stages of the
facility life-cycle.”
The oddness of multiple threads of governance and
responsibility is further reinforced where the CORF
and Head, LFO roles defined. I know some of the
history here, so don’t mean to bug you, but the lack
of connection between these two roles is odd.
Shouldn’t – LFO report to CORF, not BFA?
Diagram 2.1.6-18 – indicates the rich complexity.
No change.
The LFO deals with business-related policy,
process, and procedures for oversight of
major facilities and is properly located in
BFA reporting to the CFO. Where the CORF
deals with strategy issues and is located in
the Office of the Director. See Figure 2.1.6-1
for a graphic of these relationships.
“Due to the Federal appropriations
process, there may be one or
more years between the PDR and
the start of construction, which is
predicated on successful
completion of the FDR. During this
time the NSF will review the
project at least annually to ensure
that the total project cost and
basis of estimate (BOE),
acquisition strategies, schedule,
and risk management plan
presented at the PDR are still
valid.”
Delete paragraph.
Accepted.
Figure
2.1.6-4,
2.1.6-18
42.
2.3.3.2,
2.3.3-2
NSF
BFA-LFO
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 11
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
2.3.3.2,
2.3.3-3
NSF
BFA-LFO
Delete paragraph. The no-cost overrun policy is
applicable to the post FDR TPC negotiated for
award.
Accepted.
44.
2.4.2.1,
2.4.2-1
AUINRAO
Close out review – is this an internal (NSF only) or
external review?
No change.
The close-out review is organized by NSF
and comprised of panelists external to NSF.
45.
2.4.2.2,
2.4.2-1
NSF
BFA-DGA
NSF has implemented a “no cost
overrun policy” on major facility
construction projects. This policy
requires that the Total Project
Cost (TPC) estimate developed at
the Preliminary Design Stage has
adequate contingency to cover all
foreseeable risks, and that any
cost increases not covered by
contingency be accommodated by
reductions in scope.
“As part of the annual
construction review process
outlined above in section 2.4.1
Construction Award Management
and Oversight, at an appropriate
time approaching or following
construction completion, NSF will
conduct a final construction
review. This review is intended to
assess the extent to which the
required scope was delivered in
accordance with the PEP and
award terms and conditions.”
“Per the PAPPG, the Recipient may
authorize a one-time extension of
the end date of a cooperative
agreement of up to 12 months.
The Recipient should discuss such
an extension with the Program
Officer (PO).”
The section about NCE’s is incorrect. Grantee
approved NCEs are only for grants. In accordance
with the NSF CA FATCs, grantee-approved
extensions are NOT applicable to cooperative
agreements.
Accepted.
Sentence was deleted, and language revised
to state the PO recommends approval of the
first NCE.
43.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 12
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
2.4.2.2,
2.4.2-3
AUINRAO
47.
2.5.1,
2.5.1-3
AUINRAO
“It is the Recipient’s responsibility
to manage and maintain the NSFfunded facilities, equipment, and
instrumentation used in the
conduct research. However, NSF
rarely maintains ownership to
major research equipment and
facilities it funds.”
I disagree. NSF are the owners, it may be the title of
things is in the mgt org hands, but if they lose
recompetition, they are obliged to sign the
equipment to new mgt org. May need some finesse
in this description.
48.
2.5.1,
2.5.1-4
AUINRAO
“In most cases, NSF will annually
conduct Operations Reviews of its
major multi-user research
facilities, utilizing an external
panel of experts…”
Guide references Annual Operations Reviews … we
call them Program Operating Plan reviews…
49.
2.5.1,
2.5.1-3
Woods
Hole-OOI
The Annual Work Plan (AWP)
describes what the facility expects
to accomplish in the coming fiscal
year. For many facilities, the AWP,
annual operations proposal, and
Cost Estimating Plan (per Sections
4.2.2 and 4.2.4) can be combined
as one document, so long as all
elements are addressed.
Not a comment for the draft, but I would be
interested in seeing a sample document for the
layout and appendices that accomplishes this.
46.
MFG Language/Observation
Comment
NSF Response/Resolution
The descriptions through this whole section about
transitions to Ops were a bit simplistic. In reality,
there are profound interactions between
construction and operations, with potential
costs/damages going both ways depending on what
happens, complexity about ownership of equipment
(owned by construction, ops?) etc. Seems like
between “construction” section and “ops” section
there should be more discussion of several vexing
transition issues.
No change.
There are indeed many possible scenarios
that may arise during the transition from
construction to operations. The period
leading up to the close-out review is an
opportune time to identify and discuss these
types of issues. Adding these issues to the
charge to the review panel creates an
occasion for an external and unaligned
group of experts to offer recommendations
on resolution.
Accepted.
Section 2.5.1 and Section 6.6 were revised to
clarify it is the Recipient’s responsibility to
maintain the NSF-funded facilities and NSF
ownership is specified in the terms and
conditions of the award and NSF may
choose to invoke it contingent interest to
take title or transfer to another
organization.
Accepted.
The operations review referenced here is
not a formally defined term. To reflect this
concept “operations review” is not
capitalized throughout the document.
Reviews with similar objectives may have
different names.
No change.
Responded directly to commenter.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 13
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
2.5.1,
2.5.1-3
Woods
Hole-OOI
The Annual Report describes in
detail the activities of the facility
in the previous twelve months.
This report is required by NSF
policy and necessary to review
progress on that year’s
performance goals as described in
the AWP.
Accepted.
Language has been revised to clarify that the
period for the Annual Report is based on the
award date and award terms and conditions.
51.
2.6,
2.6-1
AUINRAO
“While not part of the annual
budgeting process, proposals may
be requested to address partial or
full divestment of the facility
following the award period,
including property divestment,
decommissioning, and disposition
costs and other costs related to
employee separations.”
It would be helpful to have definitive clarification of
what should be included in the “Annual” report
timeframe. This document mentions the “previous
twelve months” and the “year’s goals described as
in the AWP.” However, in reality, the Annual report
is due 90 days before the end of the AWP POP, so
the “Annual” Report either covers 12 months of two
different project years (3 months of the prior year
and 9 months of the current report year) or just the
9 months of the current report year. Neither of
which quite fit the statement in the Major Facilities
Guide. Different directorates, even different POs
within the same directorate, are inconsistent in their
interpretation of the how the annual report
timeframe is applied.
Do you want to mention that typically it is
considered that decommissioning costs for a facility
are about the same as 2-3 years of ops costs? (I
don’t know where this rule came from, and I can
think of horrible exceptions e.g. AO, but giving some
sense of scale might be considered).
52.
3.2,
3.2-1
3.3,
3.3-1
AUINRAO
AUINRAO
“NSF FACILITY PLAN [RESERVED]”
TBD
“NSF OVERSIGHT MANAGEMENT
PLANS FOR THE MAJOR FACILITY
LIFE CYCLE”
title is “NSF Oversight Management Plans” should
be “NSF Internal Management Plans” ?
50.
53.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
It is more appropriate for a review body to
assess or advise on the cost estimate range
based on each facility’s situation and
divestment plan. This guide will apply to a
wide range of facilities and projects with
very different research focuses. Setting a
range that can be applied to all may not be
realistic and could lead to more difficulties
and controversial.
No change.
Comment is an observation.
Accepted.
Page 14
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
3.4.1,
3.4.1-1
AUINRAO
Table 3.4.1-1 List of the Typical
Components of a Project
Execution Plan, with Sub-Topics
and Descriptions
Table should include any special construction
requirements, closeout requirements?
55.
3.4.1,
3.4.1-1
AUINRAO
Important for PEP to document all assumptions and
boundary conditions driving project design and
implementation. (didn’t hear that clearly…)
56.
3.4.1,
3.4.1-1
through
3.4.1-6
3.4.1,
3.4.1-1
AUINRAO
Table 3.4.1-1 List of the Typical
Components of a Project
Execution Plan, with Sub-Topics
and Descriptions
Table 3.4.1-1 List of the Typical
Components of a Project
Execution Plan, with Sub-Topics
and Descriptions
Table 3.4.1-1 List of the Typical
Components of a Project
Execution Plan, with Sub-Topics
and Descriptions
Accepted.
The second paragraph on page 3.4.1-1 was
added to address special construction
requirements.
The Table currently has component 16.1
Project Close-out Plan to document the
closeout requirements.
Accepted.
Added language to this section.
58.
3.4.1,
BOAC
Subcom
mittee
Table 3.4.1 PEP components:
Section 1.2 (Scientific
Requirements), Section 4.1
(Project Definition) and Section 4.4
(Scope Management Plan).
59.
3.4.2,
3.4.2-1 –
3.4.2-3
AUINRAO
Threshold or Non-negotiable science and or
technical performance requirements should be
tracible. There should be more clearly defined
criteria around scoping/de-scoping decisions.
Threshold or Non-negotiable requirements are the
level of requirements below which the project isn’t
worth doing.
TBD
54.
57.
AUINRAO
Table 3.4.1-1 is kinda long, really breaks up the
document. Put in as an appendix (like previous
LFMs?).
Will be assessed and addressed in a later
version.
Missing from Table 3.4.1-1 – A Construction to
Operations Transition Plan?
No change.
The administrative process for transition to
operations is addressed in Component 15.2.
The technical aspects are addressed in
Component 15.1. These components could
be in one or more plans as needed by the
specific project.
No change.
NSF will develop internal guidance for major
facility reviews to ensure that
scoping/descoping criteria and the impact of
descoping decisions on the threshold
requirements be carefully evaluated.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
Comment is an observation.
Page 15
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
3.4.2.15,
3.4.2-2
AUINRAO
“The Operational Readiness Plan
defines the process for acceptance
at the end of construction and
determining operational
readiness.”
Para 4 - Operational Readiness Review … this gets
back to the point raised above… sometimes you’re
already in operations while still in construction?
Might be good to check that your callouts here (like
this review) make sense in these complex C->O
transitions. Could be “Science Operations Readiness
Review”.
61.
3.4.2.15,
3.4.2-3
AUINRAO
Having only concept of operations doc at this stage
seems too little too late.
62.
3.5.2,
3.5.2-1
AUINRAO
A Concept of Operations Plan
(PEP-15.3) is also required by the
PEP, whereby the hand-off from
construction project responsibility
and funding to operations
responsibility and funding is
detailed.
“Federally Funded Research and
Development Centers (FFRDCs)
follow a slightly different process
and cannot be renewed or
divested until a comprehensive
review is performed.
No change.
The MFG language is referring to a “Plan”
not a “Review”. The Guide states:
“Transition from construction to operations
could be a single acceptance event or
multiple depending on the nature of the
project.”
“At least one year prior to initial
commissioning activities, the plans must be
updated and provided to NSF for review.”
“Once the commissioning planning is
complete, an operations readiness review
may be held to examine and comment on
the plan.”
Accepted.
Language revised to clarify that the Concept
of Operations Plan is refined during the
Construction Stage in preparation for the
Operations Stage.
60.
Discussion of FFRDC complexities. Don’t think
FFRDC’s are defined anywhere in this doc.
Guide talks about a “comprehensive review”; aren’t
they all?
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
Language revised to clarify FFRDC follow a
different process as outlined in the Federal
Acquisition Regulation (FAR).
Page 16
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
63.
4.2.1,
4.2.1-1
through
4.2.1-4
BOAC
Subcom
mittee
4.2.1 Overview of Guidance and
Process for Both Construction and
Operations Awards
The methodology used for estimating purposes
should be listed in this order of preference: 1)
Actual/historical data for the system/subsystems
being estimated; 2) Analogous data with
adjustments to reflect the technical and complexity
differences; 3) Parametric data should be used for
higher level WBS - modified to reflect the technical,
size, weight, quantity and/or schedule of the system
being estimated; 4) Expert opinion - used only if a
secondary methodology is used to substantiate the
expert opinion provided by the recipient or
evaluator.
Accepted.
Partially accepted. A different order of
estimating method preferences is not
currently provided anywhere in Section 4.2.
Section 4.2.2.1 and 4.2.3.4 have the
following, in order of increasing maturity
“(e.g., expert opinion, analogy, parametric,
engineering build-up, historical data)”.
4.2.3.4 also discusses expected evolution of
estimate through design phases.
Agree preferences might be helpful.
However, Analogous is considered top-down
and is less accurate than parametric. Also,
Section 4.2 applies to construction and
operations, so the focus is not solely on
systems being estimated.
Added a statement to 4.2.2.3 on GAO best
practice #6 Obtain data that:
“The best estimating method should be
chosen for each WBS element. The
following cost estimating methodologies
should be used, in order of preference, if the
data exists: (1) Actual/historical data for the
systems or operations being estimated; (2)
Detailed engineering build-up; (3)
Parametric data with adjustments to reflect
differences (e.g., technical, size, weight,
quantity, location, schedule); (4) Analogous
data with adjustments to reflect differences;
(5) Expert opinion, only if a secondary
methodology is used to substantiate.”
4.2.2.3 currently states that an explanation
for choosing a particular estimating method
should be documented in the CEP and Cost
Book.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 17
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
4.2.1 and
4.2.2,
4.2.1-1
through
4.2.2-17
BOAC
Subcom
mittee
Improve cost estimate documentation. Both the
Recipients and evaluators should clearly document
the estimate approach, quantitative justification and
support.
No change.
Section 4.2 describes how estimate shall be
documented. GAO considered this best
practice of documenting the estimate to be
“fully met”.
65.
4.2.2.1 &
Figure
4.2.2-1,
4.2.2-2 and
4.2.2-3
NSF
CISE
4.2.1 Overview of Guidance and
Process for Both Construction and
Operations Awards and
4.2.2 Elements of Both
Construction and Operations
Estimates
Figure 4.2.2-1 Sample Project
Management Control Systems
Relationship Diagram
Accepted.
Figure 4.2.2-1 was revised to add
clarifications as to what is applicable to
operations versus construction. CEP, WBS,
Cost Model Data Set and Cost Reports are
elements of both construction and
operations estimates. The other elements
are applicable to construction and major
facility upgrades.
66.
4.2.2.3,
4.2.2-6
thru 4.2.29
Woods
Hole-OOI
Application of GAO Cost Guidance
to Major Facilities.
67.
4.2.2.4
Woods
Hole-OOI
68.
4.2.2.5,
4.2.2-15
AUINRAO
“To assist Recipients in
determining the difference
between a subaward and a
contract, please refer to the
“Subrecipient vs. Contractor
Checklist,” developed by the
Association of Government
Accountants.”
“The payment of fee may be
authorized for major facility
construction and operations
awards, unless otherwise
prohibited in specific
circumstances by NSF. …”
It is unclear as to what guidance applies to the
operations stage. In many places, it seems that the
operations phase is being treated the same way as a
construction project in terms of budget, risk
management, and projects controls requirements.
A case in point is Figure 4.2.2-1 which is contained
within section 4.2.2 Elements of Both Construction
and Operations: The figure has boxes for integrated
master schedule, EVM, Schedule reports etc. that
should apply just to construction projects
The addition of this section is helpful by
summarizing each of the 12 GAO steps and
providing some interpretation, insight, and
suggested approaches as to how the steps might be
applied to the MF.
Inclusion of the Subrecipient vs. Contractor checklist
is useful. This is an area that often raises questions.
Would it be useful to open this section with an
explanation of the uses/philosophy behind Fee?
Will be assessed and addressed in later
version.
64.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
Comment is an observation and no response
required.
No change.
Comment is an observation and no response
required.
Page 18
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
4.2.2.5,
4.2.2-15
AUINRAO
In the middle, the text refers to “cost contracts”,
what is relevance for CA/CSAs?
No change.
This is the name used in the U.S. Code to
describe the percentage amount NSF will
not exceed. NSF has determined to use
code for Cooperative Agreements and
Cooperative Support Agreements.
70.
4.2.2.5,
4.2.2-15
4.2.2.6,
4.2.2-16
In the FEE paragraph, change “Cost Proposal Review
Document (CPRD)” to “Decision Memo” to align
with internal terminology.
OMB inflators … why isn’t LFO delivering each year
for all projects?
Accepted.
71.
NSF
BFADACS
AUINRAO
“The amount of fee will not
exceed the statutory limitations
pertaining to cost contracts set
forth at 41 U.S.C. 3905,
notwithstanding that the fee is
provided through a cooperative
agreement.”
72.
4.2.3.4,
4.2.3-7
AUINRAO
Figure 4.2.3-2 Construction Cost
Book Sheet Sample Format
Figure is unreadable.
73.
4.2.4,
4.2.4-1
AUINRAO
Section 4.2.4 - Additional
Guidance for Operations Estimates
No discussion of metrics? KPIs?
74.
4.2.4.4,
4.2.4-3 and
4.2.4-4
AUINRAO
Language discussing contingency in operations
estimates – is that a thing now? Has been outlawed
previously.
75.
4.2.5.1,
4.2.5-1
NSF
BFA-LFO
Figure 4.2.4-1 Operations WBS and
Budget Sample Format
“Contingency, if requested, must
be in compliance with Section
4.2.6 of this Guide.”
69.
“Recipients are not limited to
using the publicly available
economic assumptions and broad
OMB inflators
(https://www.whitehouse.gov/om
b/budget/Supplemental) when
doing cost estimates.”
Management reserved, if authorized, is held by NSF
and not a portion of the Recipient’s risk-adjusted
TPC.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
The information on OMB inflators was
removed since the practice of NSF providing
them for all projects is no longer followed or
necessary. OMB and the Congressional
Budget Office provide publicly available
economic assumptions. Specialized
escalation data may also be more helpful for
a particular facility.
Will be assessed and addressed in later
version.
Will be assessed and addressed in later
version.
Metrics may also be more appropriately
discussed in another section.
No change.
LFM language has existed in section 4.2.6.
Accepted.
Page 19
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
4.2.5.2,
4.2.5-2
AUINRAO
“The scope contingency should be
well considered and strive to
minimize negative impacts.”
77.
4.2.5.2,
4.2.5-2
AUINRAO
No change.
Sentence above states “… the project’s
prioritized and time-phased de-scoping plan
should equal at least 10% of the baseline
budget…”
No change.
NSF Policy is 10% of baseline budget.
78.
4.2.5.2,
4.2.5-2
AUINRAO
Good to make the relevant Directorate on the hook
for the first 10%
No change.
Comment is an observation and no response
required.
79.
4.2.5.2,
4.2.5-3
AUINRAO
Delete “even when properly managed” weird
Accepted.
80.
4.2.5.8,
4.2.5-10
AUINRAO
“...the project’s prioritized and
time-phased de-scoping plan
should equal at least 10% of the
baseline budget...”
“NSF does not normally hold a
management reserve for a specific
project as part of the TPC. As a
result, the Directorate is
responsible for the first 10% of
costs which exceed the authorized
TPC.”
“Since development of
contingency is statistically-based,
there is a chance that not every
risk will be realized at its
maximum impact. Therefore, even
when properly managed, it is
possible that contingency dollars
will remain at the end of the
project.”
Each project in construction must
report monthly to NSF on the
status of the project, while
projects in the Design stage are
highly encouraged to submit a
monthly report
“Scope Contingency” is mentioned. Ok, but key
point is it’s hard to use late in the project… add
some text to indicate that scope management
plans/docs must be accompanied by time profile
information?
Text says scope plan should be for at least 10% of
baseline budget – you mean 10% of TPC?
Reporting also impacted by construction to
operations transitions … may be need for additional
reporting in overlap period.
No change.
Section 4.6.2 reference award terms and
conditions (T&Cs).
Reporting to NSF is defined in the governing
CA/CSA and is applicable through the award
end date. In cases where there are
construction and operations CSAs in place,
reporting is required for each award as
defined in the governing terms and
conditions.
76.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 20
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
4.2.5.5,
4.2.5-12
AUINRAO
Fig 4.2.5-1 hard to read…
Will be assessed and addressed in later
version.
82.
4.2.6,
4.2.6-1
AUINRAO
4.2.6,
4.2.6-1
Woods
Hole-OOI
84.
4.4,
4.4-1
AUINRAO
Again referring to contingency in ops. If we’re doing
this – what happens to unallocated contingency in
Ops – roll to next year?
Need to correct typos/grammar: Text should read:
“there are many inherent risks with operations” or
“there is inherent risk with operations” and “Basis of
Estimate”
Para 3 talks about “Transition Plan” … defined
previously?
No change.
LFM language has existed in this section.
83.
Figure 4.2.5-1 Sample of a Change
Control Request Form, with
instructions for filling out the
various sections
Section 4.2.6 Budget Contingency
Planning during the Operations
Stage
Budget Contingency Planning
during the Operations
85.
4.4,
4.4-1 and
4.4-2
AUINRAO
To list of bullets … add “discussion of risks to project
from operations delays”?
Accepted.
86.
4.6.3.3
4.6.3-2
Woods
Hole-OOI
Changing the five-year cycle to “a regular review
cycle” makes the statement vague and opens up the
possibility of more frequent BSRs. If the intent is to
delete the five-year requirement (as is noted in #25
in the Summary of Significant Changes), perhaps the
language should be the same as the referenced BSR
Guide (although the posted version is old). The
Guides identifies the frequency as “Whenever
possible, the BSRs are conducted on a five-year
cycle…”
No change.
The intent is to delete the five-year cycle
since the frequency is informed by the
annual risk assessment. With this new
framework it is feasible for NSF to conduct
more frequent BSRs which align with the
identified risk rather than a fixed time cycle.
The BSR Guide is under revision to reflect
the new framework and annual risk
assessment.
81.
“Elements of the Transition Plan
are first addressed during
Conceptual Design, and become
progressively more detailed as
planning evolves. During
construction, the PO reviews the
plan, utilizing internal staff,
external experts, consultants,
external review panels and the
resources of the Large Facilities
Office.”
“The review of the plans for
commissioning and acceptance
should consider the following
questions: ...”
“BSRs are conducted on a regular
review cycle which is informed by
the internal NSF annual Major
Facility Portfolio Risk Assessment.”
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
Accepted.
Language deleted “Transition Plan” to align
with terminology used in the rest of the
guide, specifically 3.4.1 and 3.4.2.
Page 21
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Source
MFG Language/Observation
Comment
NSF Response/Resolution
4.6.3.6,
4.6.3-5 and
4.6.3-6
4.6.4,
4.6.4-1
AUINRAO
Section 4.6.3.6 Earned Value
Management Verification,
Acceptance, and Surveillance
“The NSF performance metric goal
(using EVM) is keeping both the
total project cost and schedule
variances against the Performance
Measurement Baseline at, or
better than, negative 10 percent.”
Refer to section 6.8?
Accepted.
Positive EVM Variances are important as they
characterize the accuracy of the baseline estimate,
so suggest changing this: "at, or better than,
negative 10 percent," to "at, or better than, positive
or negative 10 percent."
89.
4.6.4,
4.6.4-1
Woods
Hole-OOI
"Projects that are less than 10
percent complete are not held to
this goal because EVM data is less
meaningful statistically in the very
early stages of a project"
90.
4.7.2,
4.7.2-1
AUINRAO
Section 4.7.2 Partnership Funding
91.
5,
5-1 thru
5-4
Woods
Hole-OOI
Section 5 - GUIDANCE FOR MIDSCALE RESEARCH
INFRASTRUCTURE PROJECTS
This statement is not correct. EVM in the earliest
stages of a project are actually good indicators of
the quality of the baseline budget and schedule. For
example, statistics have shown that a project that is
less than 10 percent underway, but has significant
cost and schedule variances, is likely to never
recover.
Suggest you structure these agreements to produce
a default operations number, consistent with a 5-yr
plan. (ALMA is suffering from lack of this). Or a
default that this year is last years plus the
appropriate inflation.
Overall comment: This section is much less detailed
than others, although it does refer to other sections
as reference. There are a wide variety of projects
that will fit into this category, and while it is
preferable to correlate the tracking and reporting
overhead to the size of the project, the sections
should list minimum requirements, and use
established project management protocols to define
the difference in management scope between a mid
and large scale projects.
No change.
The MFG requires Recipients to report both
positive and negative cost and schedule
variances >10% (see last paragraph of this
Section and Section 4.6.2)
The negative 10% in the first paragraph of
this Section is referring to NSF’s
performance metric per GPRAMA and OMB
requirements.
No change.
This is applicable to NSF’s performance
metric per GPRAMA. This is not applicable
to Recipient reporting to NSF.
Recipient earned value reporting
requirements are in Section 4.6.2.
87.
88.
Sec Para,
page no.
Woods
Hole-OOI
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
The comment is a Memorandum of
Understanding (MOU) level of detail and not
appropriate for the MFG.
No change.
Mid-Scale projects are defined in Section 1.4
of this document which is referenced in the
first sentence of this section. The specific
circumstances of each Mid-Scale project will
be unique, as are larger MRFEC projects, and
thus the oversight requirements need to be
tailored through discussions between NSF
and the Recipient. In addition, if there are
minimum requirements, they will be
detailed in the proposal solicitation.
Page 22
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
5,
5-1
Woods
Hole-OOI
Section 5 - GUIDANCE FOR MIDSCALE RESEARCH
INFRASTRUCTURE PROJECTS
"does not preclude the requirement for appropriate
rigor": Add detail to the remaining sub-sections that
support this statement, similar to section 4.
93.
5,
5-1
Woods
Hole-OOI
94.
5,
5-2
Woods
Hole-OOI
change "Level of maturity on collaboration or
partnerships" to "Level of maturity of collaborations
or partnerships."
change "Technical feasibility and consideration of
risks" to "Technical feasibility, cost impacts, and
consideration of risks and opportunities."
95.
5,
5-2
Woods
Hole-OOI
“...some typical project
characteristics that may be
considered...”
“Mid-scale projects are selected
based on the merit review criteria
detailed in the program
solicitation. However, some typical
project characteristics that may be
considered ...”
“Mid-scale projects are not subject
to the formal stage-gate review
process given in Section 2.3 of this
Guide. However, the internal
proposal review process used by
NSF for construction/acquisition
proposals should be sufficiently
robust to assess readiness ...”
No change.
Mid-scale projects can take many forms
from unique configurations of commercially
procured equipment to projects requiring
state of the art, not yet available technology.
Thus, the oversight requirements for these
projects need to remain flexible so an
optimal solution can be tailored through
discussions between NSF and the Recipient.
Accepted.
92.
"not subject to the formal stage-gate review
process", and the "internal proposal review... should
be sufficiently robust": The "internal review" should
list types of minimum criteria, and who within NSF
conducts (and approves) the reviews. The NSF
Program Officer?
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
The following bullet was added to reflect
that the that the proposal budget itself and
life cycle costs may be considered: “Budget
alignment with solicitation and lifecycle cost
impacts.”
No change.
The minimum criteria for reviews of Midscale proposals will be detailed in the
specific solicitation. The question of who
within NSF conducts and approves the
reviews depends on which office initiates
the solicitation. Sometimes, solicitations are
initiated by a specific Directorate or Division
and others can be NSF-wide solicitations.
The review process and approval authority
may differ from one solicitation to another.
Page 23
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
5,
5-3
Woods
Hole-OOI
“Although substantial rigor is
required in establishing the TPC,
mid-scale research infrastructure
projects are not subject to NSF’s
“No-Cost Overrun” policy used for
major facilities projects.”
"not subject to NSF's "No cost overrun" policy": Add
formality to this statement, certainly add detail to
section 4.3 (especially critical as other reporting /
tracking mechanisms are being simplified).
97.
5,
5-4
NSF
BFA-DGA
“LFO Liaison to assist Programs
and the Grants Officers in the
appropriate BFA unit (either the
Division of Grants and Agreements
(DGA) or Division of Acquisition
and Cooperative Support (DACS))
depending on the funding
program.”
Revise to: “LFO liaison to assist Programs and the
Grants and Agreements Officers in the appropriate
BFA unit (either Division of Grants and Agreements
(DGA) or Division of Acquisition and Cooperative
Support (DACS)) depending on the funding
program.”
98.
6.2,
6.2.1-1
through
6.2.12-1
AUINRAO
Section 6.2 Risk Management
Guidelines for Construction Stage
Risk section (6) … long. It’s good, but I wonder if
shouldn’t be in a separate document.
99.
6.2.11,
6.2.11-1
through
6.2.11-5
AUINRAO
Section 6.2.11 Contingency
Management for Risk Mitigation
Any guidelines on what year dollars the contingency
is stored in… then year? Do they inflate year to year
when carried over?
Accepted.
Added a reference to Section 1.4.5 for
definition of TPC. Revised sentence as
follows:
“Although substantial rigor is required in
establishing the Total Project Cost (TPC),
projects are not subject to NSF’s “No‐Cost
Overrun” policy used for major facilities
projects as defined in Section 1.4.5.”
Accepted.
For clarification, revised to: “LFO liaison to
assist Programs and the Grants and
Agreements Officers in the appropriate BFA
unit, either Division of Grants and
Agreements (DGA) or Division of Acquisition
and Cooperative Support (DACS), depending
on the nature of the project and the funding
program.”
Will be assessed and addressed in later
version.
NSF has equal concerns about the MFG
growing in size and intend to pare down
some sections as we enhance others. We
may bring in a professional editor as part of
a future revision to help with the
organization and areas of duplication.
Will be assessed and addressed in later
version.
Just like the initial NSB approved Project
Baseline budget, contingency budget is part
of the TPC approved by the NSB. See
Section 6.2.8.8. Additional words will be
considered as part of a later version.
96.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 24
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
100.
101.
102.
103.
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
6.3,
6.3.1-1
through
6.3.5-2
AUINRAO
Section 6.3 Guidelines for CyberSecurity of NSF’s Major Facilities
6.3,
6.3.1-1
through
6.3.5-2
6.3.1,
6.3.1-1
AUINRAO
(Beasley)
Section 6.3 Guidelines for CyberSecurity of NSF’s Major Facilities
IT/Cybersecurity guys reviewed the section. No
significant comments, although they noted that
several definitions were soft (e.g. cybersecurity
component of IT budgets 3-12% … deciding what to
include/not means you can land on any number).
The cybersecurity section lacks definition in many
areas but was too prescriptive/detailed in others.
Just a data point.
Trusted
CI
“It is necessarily a living program
that adapts, adjusts, and
advances. As such, cybersecurity
programs require reporting,
evaluation, and updating of the
program as appropriate.”
6.3.1 Introduction
“Data creation, sharing, and
analysis are central to the progress
of science. As information...”
No change.
Comment is an observation. Example text
provided, as well as other phrasing in this
section, is planned to be re-examined in a
future version.
No change.
Comment is a general observation. Section is
expected to undergo revisions in future
versions.
Accepted.
6.3.1,
6.3.1-1
AURANSO
(Cross)
Suggested change: Last sentence - strike “of the
program”
Justification: redundant and awkward phrasing
Agreeable introduction and setup for the content of
the entirety of Section 6.3.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
Comment is an observation. This Section is
planned to be addressed in a future version.
Page 25
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
104.
Sec Para,
page no.
6.3.2,
6.3.2-1 and
6.3.2-2
Source
MFG Language/Observation
Comment
NSF Response/Resolution
AURANSO
(Cross)
6.3.2 Major Facility Cybersecurity
Program
“Uniform Guidance §200.303
states that the Recipient’s internal
controls, including technology
infrastructure and security
management, should be compliant
with guidance published by the
Comptroller General or Committee
of Sponsoring Organizations of the
Treadway Commission (COSO). ...
The three pillars of a cybersecurity
program rely on a project-specific
inventory of “information assets”
to be protected. Risk-based
approaches to protection of
information assets are further
determined by a project-tailored
“information classification” which
recognizes varying degrees of
value, priority, and/or sensitivity
of the information assets. The
information asset inventory and
information classification are
described in the Controls section.”
My interpretation of the MFG is that it is inherently
directed towards NSF supported Large Facilities
(LFs). I believe LFs are all sizeable programs with
complex Information Technology Infrastructures. As
such, I feel the NSF’s commentary here provides too
much leeway to the LFs to define the size and scope
of their information security program. In this case, I
expect wide-ranging implementations of security
practices across all NSF LFs. As a result, an example
of cybersecurity difficulty would be requesting all
LFs to deploy a critical security requirement. Should
a security change be mandated, impacts of
deploying the changes will vary greatly across the
LFs, and could potentially cease work.
As a real-world comparison, within Department of
Defense (DoD) Programs, regardless of size
(whether a single laptop, or thousands of computer
systems), all programs must follow similar security
program pillars – governance, resources and
controls. These pillars are well-defined and provide
program contractors specific information to
implement these pillars, following detailed rules
(not guidance), leaving little interpretation to the
contractor. Within DoD Programs, information
security programs will have little variance, and
blanket modifications can be implemented with
consistent impact across programs.
Will be assessed and addressed in a later
version.
Your interpretation is correct that the MFG
is directed towards NSF funded major
facilities (formerly called Large Facilities) as
noted in Section 1.1 Purpose and Scope of
the MFG.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 26
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
105.
6.3.2
Trusted
CI
“Uniform Guidance §200.303
states that the Recipient’s internal
controls, including technology
infrastructure and security
management, should be compliant
with guidance published by the
Comptroller General or Committee
of Sponsoring Organizations of the
Treadway Commission (COSO)1.
Further, the Cooperative
Agreement Supplemental Financial
& Administrative Terms and
Conditions (CA-FATC) for
Recipients of Major Facilities or
Federally Funded Research and
Development Centers (FFRDC)2
requires an information security
program3 and identifies a modest
set of required components for
the program. Additionally, an
information security plan is a
required element of the Project
Execution Plan (PEP) per Section
3.4 of this Guide.”
Suggested paragraph replacement text:
A cybersecurity plan is a required element of the
Project Execution Plan (PEP) per Section 3.4 of this
Guide. Additionally, based on Uniform Guidance
§200.303, to the extent the award recipient’s IT
infrastructure is integral to internal controls, the
relevant portion of the cybersecurity program
should be compliant with guidance published by the
Comptroller General or Committee of Sponsoring
Organizations of the Treadway Commission (COSO).
Further, the Cooperative Agreement Supplemental
Financial & Administrative Terms and Conditions
(CA-FATC) for Recipients of Major Facilities or
Federally Funded Research and Development
Centers (FFRDC) requires an information security
program and identifies a modest set of required
components for the program. [add footnote
references where appropriate]
Discussion: The first paragraph is confusing since it
is an amalgam of requirements from different
sources with different scopes. We suggest moving
the sentence with the broadest scope (the
requirement for the PEP to include a cybersecurity
plan) to the start of the paragraph. Next would be
the requirement on the internal controls but
reworded to narrow applicability to cases when
internal controls implemented through information
technology. Finally, close with the Cooperative
Agreement Supplement(s). Note: Uniform Guidance
§200.303 does not actually include the phrase
“including technology infrastructure and security
management”.
No change.
The name change from “Large Facility” to
“Major Facility” is to align with the American
Innovation and Competitiveness Act (AICA)
terminology as noted in the draft Major
Facilities Guide (MFG). The MFG does not
apply to any additional facilities than the
Large Facilities Manual did. The portfolio of
“Major Facilities” is the same as “Large
Facilities”.
The paragraph is structure based on the
governing hierarchy of requirements starting
with the Uniform Guidance. Section 1.2 of
the MFG lists the hierarchy of documents
from which MFG requirements flow from.
Also, a PEP is required only for the
construction stage of major facilities. The
PEP is not required for facilities in
operations, but parts of the PEP will evolve
during construction and may be maintained
during the operations stage.
Justification: The document now applies to more
than Large Facilities or FFRDCs, so it adds clarity to
state the requirements in order of scope. Also,
clarifying the application of 200.303 to IT
implementations of internal controls.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 27
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
106.
107.
Sec Para,
page no.
6.3.2,
6.3.2-1
6.3.2,
6.3.2-1
Source
MFG Language/Observation
Comment
NSF Response/Resolution
Trusted
CI
“The three pillars of a
cybersecurity program which rest
on this foundation are
governance; resources; and
controls. Like other facility project
components, the cybersecurity
program should be appropriately
represented in standard project
documents and NSF oversight
activities such as the project
execution plan, project risk
management plan, project budget,
project reports, and project
reviews.”
Suggest changing the sentence “The three pillars of
a cybersecurity program which rest on this
foundation are governance; resources; and
controls.” To read “The four pillars of a
cybersecurity program which rest on this foundation
are mission alignment, governance; resources; and
controls.
Will be assessed and addressed in a later
version.
Trusted
CI
“The following sections define and
describe a suggested framework
for the facility cybersecurity plan.
This framework is based on the
previously mentioned three pillars
of information security programs:
Governance, Resources, and
Controls.4 Major Facilities may use
these pillars as a framework for
founding, operating, evaluating,
and improving their information
security programs, and meeting
the award terms and conditions.”
Discussion: While the “research mission and goals of
the facility” are foundational, the actual alignment
of the cybersecurity program is an additional pillar
because the program elements there need to evolve
in concert with the other pillars.
Justification: Adding the Mission alignment pillar will
be consistent with the upcoming Trusted CI
Framework.
Suggest changing the sentence: “This framework is
based on the previously mentioned three pillars of
information security programs: Governance,
Resources, and Controls.” To read: “This framework
is based on the previously mentioned four pillars of
cybersecurity programs: Mission Alignment,
Governance, Resources, and Controls.”
Will be assessed and addressed in a later
version.
Discussion: Alignment with changes suggested for
paragraph 2
Justification: Consistent changes
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 28
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
108.
109.
Sec Para,
page no.
6.3.2,
6.3.2-1 &
6.3.2-2
6.3.3 and
6.3.3.1,
6.3.3-1
Source
MFG Language/Observation
Comment
NSF Response/Resolution
Trusted
CI
“The three pillars of a
cybersecurity program rely on a
project-specific inventory of
“information assets” to be
protected. Risk-based approaches
to protection of information assets
are further determined by a
project-tailored “information
classification” which recognizes
varying degrees of value, priority,
and/or sensitivity of the
information assets. The
information asset inventory and
information classification are
described in the Controls section.”
Suggest inserting a new page formatting command
Will be assessed and addressed in a later
version.
AURANSO
(Cross)
6.3.3 Governance
“Recommended governance
elements ... and management.”
“6.3.3.1 Roles and Responsibilities
“Successful cybersecurity
programs require an active role for
facility leadership ... risk
management decisions by facility
leadership and information asset
owners.”
Suggest changing the sentence: “The three pillars of
a cybersecurity program rely on a project-specific
inventory of “information assets” to be protected.”
To read:
“6.3.3 Mission Alignment
The other three pillars of a cybersecurity program
rely on a project-specific inventory of “information
assets” to be protected.”
Note: Requires changing the numbering of
subsequent sections and updating page
headers/footers
Discussion: Add the Mission Alignment pillar
Justification: See above
While these recommendations and definitions are
accurate and positive, I feel this is guidance (not
mandates) that leaves these recommendations as
options, open to acceptance or rejection by the LFs.
I would prefer to see that each LF is mandated by
the NSF to fill these roles and responsibilities, due to
their importance to an overall security program.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
Comment is an observation and opinion on
the generalized issue of mandating
requirements vs. flexibility for major
facilities to scope their own cybersecurity
program. With the wide range of differences
between NSF major facilities in both science
community and management structures,
NSF feels flexibility is appropriate in this
version of the MFG.
Page 29
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
110.
Sec Para,
page no.
6.3.3.1,
6.3.3-1
Source
MFG Language/Observation
Comment
NSF Response/Resolution
Trusted
CI
“In addition, most cybersecurity
programs identify a senior security
role, such as a Chief Information
Security Officers (CISO) or
Information Security Officer (ISO)
as owner of the cybersecurity
program and lead decisionmaker
for operational aspects of the
cybersecurity program.”
Suggest changing: “In addition, most cybersecurity
programs identify a senior security role …” To read:
“In addition, cybersecurity programs should have an
identified senior security role …”
Accepted.
111.
6.3.3.2,
6.3.3-1 and
6.3.3-2
AURANSO
(Cross)
112.
6.3.3.3,
6.3.3-2
AURANSO
(Cross)
6.3.3.2 Policies
“Every facility project with
information assets will require the
development, approval and
implementation of some
information security policies
within its cybersecurity program.
...”
6.3.3.3 Risk Management and
Acceptance
“Cybersecurity programs employ a
risk-based approach to
information security and as such,
there is inherent
acknowledgement by NSF that any
valuable activity requires
acceptance of residual risk. ...”
Discussion: Having an individual responsible for the
cybersecurity program is important and should not
be an undue burden. The task is not necessarily fulltime but the core responsibility for the program
should be centralized.
Justification: Strengthen the guidance to have
individual primary program responsibility
A stronger message would be for the NSF to adopt
Trusted CI policy templates, and provide them
directly from NSF accessible resources.
I am not confident LF leadership will acknowledge
the importance of Risk Management and
Acceptance of their security programs based on the
commentary in section 6.3.3.3. I feel this important
task could easily be ignored, or set aside, due to the
lack of any NSF published requirement. LFs
leadership is focused on their scientific project, and
if they are not required to accomplish an
information technology security related task, I
would expect little effort and few resources being
available to LF Cybersecurity/Information
Technology teams for these types of activities.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Will be assessed and addressed in a later
version.
NSF major facilities currently have access to
Trusted CI project resources and
documentation currently as noted in Section
6.3.3.2.
No change.
Comment is an observation and opinion on
the generalized issue of mandating
requirements vs. flexibility for major
facilities to scope their own cybersecurity
program. With the wide range of differences
between NSF major facilities in both science
community and management structures,
NSF feels flexibility is appropriate in this
version of the MFG.
Page 30
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
113.
114.
Sec Para,
page no.
6.3.3.3,
6.3.3-2
6.3.3.4,
6.3.3-2
Source
MFG Language/Observation
Comment
NSF Response/Resolution
Trusted
CI
“In addition to the Center for
Trustworthy Scientific
Cyberinfrastructure (CTSC) guide
which is tailored to the scientific
community, ...”
Suggest changing: “Center for Trustworthy Scientific
Cyberinfrastructure (CTSC)” To read: “Trusted CI”
Accepted.
Trusted
CI
“Given the dynamic technology
and cybersecurity landscape,
organizations are advised to plan
for periodic evaluations of the
cybersecurity program, including
policies, practices, and controls.
While project management and
NSF oversight will involve regular
reporting and review of program
milestones, outcome metrics, and
incidents, the project is
encouraged to consider periodic
self-assessments, external or
stakeholder reviews, and
evaluation of incident response.
Tools are available from variety of
sources to aid in assessment.”
Discussion: CTSC has changed its name to Trusted
CI.
Justification: Update organization name
Suggest changing: “... organizations are advised to
plan for …” To read: “ … organizations should plan
for …”
Suggest changing: “ … the project is encouraged to
consider …” To read: “... the project should include
in the NSF review …”
Accepted.
Will be assessed and addressed in a later
version.
Discussion: Given that NSF oversight will require a
review of the cybersecurity program, the language
in this paragraph should be strengthened.
Justification: Ensure the cybersecurity program
undergoes periodic evaluation and review
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 31
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
115.
116.
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
6.3.3.4,
6.3.3-2
AURANSO
(Cross)
6.3.3.4 Evaluation
“Given the dynamic technology
and cybersecurity landscape,
organizations are advised to plan
for periodic evaluations of the
cybersecurity program, including
policies, practices, and controls.
...”
No change.
Recipients are required to establish and
maintain a cybersecurity program in
accordance with the award terms and
conditions for compliance with Uniform
Guidance. If an NSF oversite exercise were
conducted, the expectation is that the
Recipient is following their established and
documented program.
6.3.4.1 and
6.3.4.2,
6.3.4-1
AURANSO
(Cross)
Section 6.3.4 Resources
Subsections 6.3.4.1 Budget and
6.3.4.2 Personnel
I like the tone of this section. It does tend to conflict
with the remainder of the overall section 6.3. All of
section 6.3 is providing guidance of some sort, but
does not prepare or setup the LF for NSF oversight:
security evidence/reports, review of milestones,
metrics, incidents, etc. These are tangible NSF
actions that I feel an LF cannot appropriately
prepare for without specific actions, controls, etc.
for the LF to fulfill. Simple example: If the NSF does
not require minimum password complexity, how
can an LF define password complexity to be
compliant with NSF expectations during an NSF
oversite exercise?
I do not understand the value of this section, and I
believe it will be less useful to LF leadership. This
may be a stretch, but I feel including comments
about how the LF is required to include
Cybersecurity funding line-items/work breakdown
structure codes will have more impact. My feeling as
an IT professional is that these funds should be
included as separate/called-out “bucket” within all
LF Cooperative Agreements between the NSF and
the LF. Cybersecurity seems as-needed as every
other aspect of the LF’s construction and
operations, so I would ask why are Cybersecurity
funds not outlined and agreed upon during initial LF
support by the NSF? The current Cooperative
Agreement Supplemental Financial & Administrative
Terms and Conditions (CA-FATC) for Recipients of
Major Facilities or Federally Funded Research and
Development Centers (FFRDC) are far too vague for
LFs to implement security programs that the NSF (or
outside security consultant) could review
comparatively.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
As noted in Section 6.3.2, there are various
considerations in developing and
maintaining a cybersecurity program
including a budget. The MFG gives major
facilities the flexibility to balance the mission
and goals with good business practices.
Page 32
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
117.
118.
Sec Para,
page no.
6.3.4.2,
6.3.4-1
6.3.5
6.3.5-1 and
6.3.5-2
Source
MFG Language/Observation
Comment
NSF Response/Resolution
Trusted
CI
“In addition to technical skills,
teaching skills, communication
skills and negotiating skills are
endemic to cybersecurity
programs and are, therefore,
necessary personnel
considerations.”
Suggest changing: “In addition to technical skills…”
To read: “While technical skills are important …”
Accepted.
Section 6.3.5 Controls
Subsections 6.3.5.1 Information
Asset Inventory, 6.3.5.2
Information Classification and
6.3.5.3 Control Set
I like calling out these controls, assets and
information classification guidelines. This section,
6.3.5, is the heart of what a LF Chief Information
Security Officer would be looking for from the NSF
to ensure their LF was maintaining a proper
Cybersecurity program, with respect to the NSF. In
my position, I would hope there would be a control
set requirement guide, with specific and detailed
controls, that mandated LF’s compliance. I believe
stronger wording, with accountability, would move
LFs toward better overall security posture. Even if
the NSF does not now have a mechanism to be the
Cybersecurity “Police”, having the requirements in
place for LFs to implement would be extremely
beneficial. This would arm LF Cybersecurity and
Information Technology professionals with a
minimum set of security controls to deploy across
the LF’s infrastructure. In this scenario, LFs could
collaborate within a cybersecurity control set
‘language’ that is recognizable across the entire NSF
landscape. Currently, LFs can only discuss how to
deploy certain technologies. As it stands now, LFs
cannot help one another with deploying specific
technologies and mechanisms to ensure their
information security program controls are meeting
NSF defined/blessed Cybersecurity settings and
requirements.
AURANSO
(Cross)
Discussion: The sentence is easily misread due to
the comma-separated list.
Justification: Better separation of “technical skills”
from the other listed items
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
Comment is an observation.
Page 33
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
119.
120.
Sec Para,
page no.
6.3.5,
6.3.5-1
6.3.5.1 and
6.3.5.2,
6.3.5-1 and
6.3.5-2
Source
MFG Language/Observation
Comment
NSF Response/Resolution
Trusted
CI
“Information security objectives,
and controls, center on
confidentiality, integrity and
availability. Controls are designed
to the facility’s portfolio of
information assets and aligned to
the corresponding information
classification for those information
assets.”
Suggested paragraph replacement text: “Controls
are tailored to the facility’s portfolio of information
assets and aligned to protect confidentiality,
integrity, and availability based on the
corresponding information classification for those
information assets.”
Accepted.
Trusted
CI
Section 6.3.5.1 “Information Asset
Inventory” and 6.3.5.2
“Information Classification”
Discussion: The paragraph is poorly worded or
contains redundant information.
Justification: Better wording for the point being
made.
Suggested change: Move the two sections under the
Mission Alignment pillar and renumber the Control
Set section. Make appropriate page header/footer
alterations.
Will be assessed and addressed in a later
version.
Discussion: The subsections now belong under
Mission Alignment and should be moved entirely
under that pillar.
121.
6.3.5.1,
6.3.5-1
AUINRAO
122.
6.3.5.1,
6.3.5-1 and
6.3.5-2
AUINRAO
Section 6.3.5.1 Information Asset
Inventory – “Organizational
identification and location of
information assets is a
prerequisite to competently
securing those assets. See, CIS
Critical Security Controls 1 and 2,1
and NIST 800-53 control CM-8.
Section 6.3.5.1 Information Asset
Inventory
Justification: These topics are part of the Mission
Alignment pillar.
IT here claims that NIST 800-171 is the more
appropriate reference.
Definition of “Information Systems” .. the scope you
suggest is “insane”, according to my guys; SCADA
and HVAC? Too deep. Need layered security.
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
No change.
NIST 800-171 is a subset of NIST 800-53 and
focused on confidentiality. The existing
reference is merely meant to reinforce the
importance of information asset inventory.
Accepted.
Last sentence is removed – that text does
not appear in the U.S. Code 3502 under the
definition of an “information system”.
Page 34
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Source
MFG Language/Observation
Comment
NSF Response/Resolution
123.
Sec Para,
page no.
6.5,
6.5-1 and
6.5-2
AUINRAO
Section 6.5 Environmental
Considerations in Major Facility
Planning
Seems pretty short for what is what of the most
detailed and painful parts of NSF construction
planning.
124.
6.8,
6.8-1
through
6.8-3
AUINRAO
Section 6.8 Guidelines for Earned
Value Management Systems
Doesn’t talk about the formal requirements for
EVMS mentioned earlier? sect 4.6.3.6
125.
6.8,
6.8-1
Woods
Hole-OOI
EVMS can provide metrics and uncover problems, so
change "forecast" to "uncover." Remove the word
"also" from last sentence.
126.
6.8.1,
6.8-1
Woods
Hole-OOI
“NSF recognizes that a properly
implemented Earned Value
Management System (EVMS) can
provide accurate and reliable
performance measurement
metrics and forecast potential
problems to support sound and
timely management decisions. A
properly implemented EVMS is
also essential to inform NSF’s
oversight of the project.”
“Plan all the project’s work scope
to completion.”
No change.
Environmental regulations governing NSF
construction are basically the same as for
other Federal Agencies. As noted,
determining the level of compliance
activities is complex and POs are to consult
with OGC.
No change.
The verification, acceptance, and
surveillance conducted by NSF is the last
three bullets of Section 6.8.3 and references
Section 4.6.3.6.
Accepted.
The language was revised to include
forecasting and uncover potential problems.
EVMS is a project control tool for the Project
Management Team (Recipient) and the
monthly earned value results inform NSF.
127.
6.8.1,
6.8-1
Woods
Hole-OOI
“Break down the project work
scope into finite pieces that are
assigned to a responsible person
or organization for control of
technical, schedule and cost
objectives.”
The first bullet appears to disallow the use of
"Planning Packages," which are commonly used for
activities with known work content but without
detailed schedule activities. Is this correct?
Suggest replace the word "control" with "delivery"
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
Language was added to clarify the use of
planning packages is acceptable.
No change.
The responsible person is the control
account manager for delivery of the finite
piece of work; therefore “control” of the
objectives associated with the scope is
appropriate.
Page 35
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
6.8.1,
6.8-1
Woods
Hole-OOI
“Use actual costs incurred and
recorded in accomplishing the
work performed.”
Suggest replace the word "accomplishing" with
"calculating"
129.
6.8.1,
6.8-1
Woods
Hole-OOI
Suggest replace the word "performance" with
"package"
130.
6.8.1,
6.8-1
Woods
Hole-OOI
131.
6.8.2,
6.8-2
Woods
Hole-OOI
132.
6.8.3,
6.8-3
Woods
Hole-OOI
“Objectively assess
accomplishments at the work
performance level.”
“Analyze significant variances from
the plan, forecast impacts, and
prepare an estimate at completion
based on performance to date and
the remaining work to be
performed.”
“...EVMS should be implemented
in a way that can provide the
Recipient management team a
reliable basis for objectively
assessing performance against
plan...”
Section "Guidelines for
Establishing an EVMS."
No change.
Principle is as outlined in EIA-748. Typically,
actual costs should not be the source to
determine the earned value (work
performed).
No change.
Principle is as outlined in EIA-748.
133.
7,
7-1
Woods
Hole-OOI
Guide to the Project Management Body of
Knowledge is currently on 6th Edition, 2017.
134.
8,
8-1
Woods
Hole-OOI
References: “Guide to the Project
Management Body of Knowledge,
(PMBOK® Guide), Project
Management Institute, 5th
Edition, 2013”
Acronyms: “CDR Conceptual
Design Review”
128.
Suggest adding "develop corrective actions,"
following "forecast impacts."
Accepted.
add "with" before "a reliable basis…"
Accepted.
Add EVMS Training to this section.
No change.
Training is not a component of EIA-748, nor
NDIA Intent Guide. Concur the Recipient’s
project management staff need to
understand the concepts of earned value.
This can be achieved through experience
and/or training.
Will be accessed and addressed in a later
version.
The changes in the 2017 PMBOK have not
been reviewed yet to assess changes to the
MFG.
No change.
The acronym “CDR” has multiple meanings.
For NSF major facilities, it is Conceptual
Design Review as defined in this Section.
The most common widespread definition of "CDR" is
"Critical Design Review," not "Conceptual Design
Review." Conceptual Design Reviews are frequently
abbreviated to "CoDR."
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Page 36
Public Comments on Major Facilities Guide (MFG) December 2016 & NSF Responses
Cmt #
Sec Para,
page no.
Source
MFG Language/Observation
Comment
NSF Response/Resolution
9.2,
9.2 through
9-15
AUINRAO
Section 9 Lexicon
Seemed like there are entries missing, and entries
for items not called out (Pessimistic Duration ..
seems like there was more pedagogical scheduling
info in here at some point). Fonts weird. Few entries
out of alphabetical order.
136.
9,
9-7
Woods
Hole-OOI
Second sentence of ICE is redundant to the first
sentence.
137.
Financial
Tool
Woods
Hole-OOI
Lexicon: “Independent Cost
Estimate Review. One of eight
types as defined by the
Government Accountability Office
(GAO) used by NSF to help validate
the Recipient’s estimate. An
Independent Cost Estimate (ICE) is
one of the eight types.”
NSF Financial Data Collection Tool
for Major Facilities 2019 - CSA
Budget Worksheet, Column E Current Expenditures
Will be assessed and addressed in a later
version.
Section 4.3 is reserved for future content on
schedule development and analysis. In
conjunction, the Lexicon will be reviewed.
The entries in italics are from Project
Management Institute, Inc lexicon as noted
in the footnote on each page of the lexicon.
Accepted.
Language was revised to clarify that there
are eight types of Independent Cost
Estimate Reviews and ICE is one of those
eight types.
135.
Blue cells in Column E of tab "CSA Budget
Worksheet" appear to be missing the proper
formulas. I believe they should be linked to the
"CSA Cost Worksheet".
Exhibit 3 - Public Comments on December 2018 Draft MFG along with associated NSF response
of 37
Accepted.
Worksheet will be revised so blue fields are
prepopulated.
Page 37
| File Type | application/pdf |
| File Title | Public Comments on NSF Major Facilities Guide Draft, December 2018 and NSF Financial Data Collection Tool |
| Subject | NSF Major Facilities Guide Draft December 2018 |
| Author | McGovern, Jean M. |
| File Modified | 2019-05-30 |
| File Created | 2019-05-30 |