Download:
pdf |
pdfPUBLIC COMMENTS ON
NSF LARGE FACILITIES MANUAL (LFM) DRAFT, MAY 2016
& NSF FINANCIAL DATA COLLECTION TOOL
WITH NSF RESPONES
Public comments received were sorted into three categories and are provided in three separate tables as follows:
Content and/or clarify language in the Large Facilities Manual, C-#
Typographic or editing in the Large Facilities Manual, T-#
Financial Data Collection Tool, F-#
Each comment is addressed with one of the following NSF Responses
Addressed in this version with an explanation of how the comment was addressed
No change with an explanation of why
Will be assessed and addressed in later version
Cover Page & Legend
Page 1 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
CONTENT AND CLARIFY COMMENTS TABLE
Page 2 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
LFM Language/ Observations
Comment
NSF Response
C-1
LFM Section/
Paragraph
0 - General
NSF
(DFM)
A general observation:
Will be assessed and addressed in
later version.
C-2
0 - General
NSF
(DACS)
Sections 2 and 3, in particular
Risk assessment and monitoring, as well as
contingency, are addressed in various sections of
the manual. While project lifecycle has a major
impact on the subjects, it may be more clear to
consolidate risk and contingency into a single
subject area in order to provide focus.
The concern is the use of the word Termination
as part of the construction stage. The new 2 CFR
200 which was approved last year standardized
all Federal definitions. Agencies cannot be at
odds with the use of the guidance. The LFM use
of the word Termination is going against what I
think it is you want to convey. The last stage of
the construction process is the Closeout of the
award not a termination.
The use of the word Termination is a negative
and means you end the award before the grantee
could finish the project. See below the
definitions.
Addressed in this version. The use of
the terms ‘termination’ and ‘closeout’
were revised to conform with 2 CFR
200 and FAR definitions. Term
definitions were added to the lexicon.
Related to Comment C-46.
§200.16 Closeout.
Closeout means the process by which the Federal
awarding agency or pass-through entity
determines that all applicable administrative
actions and all required work of the Federal
award have been completed and takes actions as
described in §200.343 Closeout.
§200.95 Termination.
Termination means the ending of a Federal
award, in whole or in part at any time prior to the
planned end of period of performance.
Content & Clarify
Page 3 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-3
LFM Section/
Paragraph
0 - General
C-4
0 - General
NSF
(GEO)
C-5
0 - General
NSF
(GEO)
C-6
0 - General
NSF
(GEO)
NSF
(GEO)
C-7
0 - General
NSF
(GEO)
C-8
0 - General
NSF
(LFO)
Content & Clarify
LFM Language/ Observations
References to “scope, Scope
Management Plan, scope
contingency, de-scope, scope
options”
Sections: 2.4.1; 2.4.3.2; 3.4.1;
4.2.5-3,4,10; 4.4.5; 5.2.3; 5.2.11;
8.2
Comment
NSF Response
I think the document could use a description of
science planning, academy reports, Advisory
Committee comments and how they shape
facilities at NSF in the introductory section (pull
up language from section 3.3)
Provide the location of the internal guidance
documents
Will be assessed and addressed in
later version.
Where is the Annual Work Plan guidance for
MREFC projects? How does the AWP for MREFC
map to the baseline maintenance processes?
I think there needs to be information on rebudget authority as a method of management
prior to contingency allocation in an operations
or MREFC effort. Re-budgeting may be different
than descope as it provides the opportunity to
continuously assess time phased scope priorities.
For operations this is critical and it should be
assessed first, prior to any contingency requests.
Re-budgeting may require de-obligation by the
Awardee and assuring that there is an acquisition
strategy/contract to support this method is
important.
Where is the guidance for the Awardee Advisory
Committees for the project? This is a lesson
learned for large subawards/subcontracts.
Clarify kinds of up-scope enhancements allowed
in SMP with examples, including parking of items
de-scoped between PDR and FDR or during
project. Clarify difference between scope =
deliverable and scope = implementation details.
No change. Internal guidance
documents are for NSF Staff and
located on collaborative portals.
Addressed in this version. Language
revised in Sections 2.5.1 and 3.5.
Will be assessed and addressed in
later version.
For construction, risk management
plan addresses “re-budgeting” as a
mitigation strategy within a work
package.
For operations, will be assessed and
address in later version.
Will be assessed and addressed in
later version.
Addressed in this revision. Various
edits in Sections 2.4.1 and 2.4.3 were
made to clarify the use of scope
options in a Scope Management Plan
(SMP).
2.3.2-2 SMP added to PEP list for PDR
Page 4 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-9
LFM Section/
Paragraph
0 - General
C-10
0 - General
NSF
(LFO)
C-11
0 - General
NSF
(OIG)
Content & Clarify
LFM Language/ Observations
NSF
(LFO)
Comment
NSF Response
See white paper “…NSF LFO Improvement
Suggestions 2016-06-23” on various suggestions
to improve NSF oversight, including changes to
the LFM
Contingency and EVM
expectations under contracts
Addressed in this version as
applicable to public. Others addressed
in internal guidance and some
suggested improvements are under
assessment.
The LFM is not clear about contingency
Addressed in this version. Also
expectations in the case of contracts.
addresses C-13 and C-22. Language in
We need some very simple "applicability"
the first paragraph page 1.1-1 was
statements in the opening sections of the LFM.
revised to indicate NSF awards
"Unless otherwise noted, the policies and
frequently take the form of
procedures in the manual apply to research
cooperative agreements, but may also
infrastructure projects regardless of the award
be made in the form of contracts.
instrument employed. Cooperative agreements
Cooperative agreements with
are governed by the Uniform Guidance. Contracts universities, consortia of universities
are governed by the
or non-profit organizations are
FAR for for-profits and by the Uniform Guidance
governed by the NSF Uniform
for non-profits and educational institutions."
Guidance and contracts are governed
I also need to know if the EVM reporting for
by the Federal Acquisition Regulations
FFPCs has contingency in the baseline for EVM
(FAR). Unless otherwise noted, the
reporting or if it is carried separately as it is for
policies and procedures in the manual
CAs. On funding, I don’t think we can hold back
apply to research infrastructure
‘contingency’ for a contract like we do for CAs.
projects regardless of the award
Contingency is allowed under UG. "Reserve" or
instrument employed. Also added on
"contingency reserve" is not...see SOG 2016-2
page 1.1-2 that departures from the
where we cover these definitions as well. Perhaps manual are considered, the Recipient
these definitions should also be included in
must provide a written justification
Section 8.2 (Lexicon)?
and NSF should document these in
the internal management plan.
… we note that the LFM is unclear as to which of
Addressed in this version. Language
the new requirements relate to large facilities in
revised to clarify throughout LFM,
operations and suggest NSF clarify this prior to
specifically Sections 2.5, 3.5, and
the LFM issuance. We also suggest ensuring that
4.2.6. LFM is also augmented with
LFM guidance for NSF staff and recipients aligns
internal guidance (SOGs).
Page 5 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-12
LFM Section/
Paragraph
0 - General
Source
NSF
(OISE)
C-13
1.1
Purpose and
Scope
NSF
(GEO/
PLR)
C-14
2.1
Purpose and
Scope
NSF
(GEO/
PLR)
Content & Clarify
LFM Language/ Observations
Comment
with NSF internal Standard Operating Guidance
(SOGs). As an example, see our comment on
section 4.2.4.1, NSF Policy Positions.
Sec. 2.1.6.4 and Appendix A –
We do request that the role of OISE in facilitating
MREFC panel roles and criteria
and coordinating NSF international research
for prioritizing MREFC projects; infrastructure be explicitly included both (i) in
Sec. 2.3 – Design Stage;
practice i.e., how LF projects are overseen and (ii)
Sec. 4.6 – Partnerships;
in the LF manual – which are of course related.
Sec. 5.5 – Environmental
For (ii), we recognize that, for the most part,
Considerations; etc.
specific divisions/offices within NSF are not called
out in the manual.
Paragraph 1: “Such awards
We have in the past and are in the queue now to
frequently take the form of
construct research stations using contract
cooperative agreements.”
mechanisms; why not include PLR and polar
Paragraph 2: “Historically, awards Research Station projects as opposed to casting
have been made for such diverse these as an unstated exception?
projects as accelerators,
telescopes, research vessels,
aircraft, and geographically
distributed but networked
observatory systems.”
“…(NSF) investments through
Include polar research stations
the … (MREFC) Account provide
state-of-the-art infrastructure
for research and education,
such as laboratory and field
instrumentation and
equipment, multi-user research
facilities, distributed
instrumentation networks and
arrays, and mobile research
platforms.”
NSF Response
No change. Comment is an
observation.
Addressed in this version. In Section
2.1.6.1, OISE is listed as part of the
Strategic Group for Integrated Project
Teams (IPT).
Addressed in this version. In Section
1.1, “Such awards frequently take the
form of cooperative agreements, but
may also be made in the form of
contracts.
Addressed in this version. In Section
2.1, “remote research station” added.
Page 6 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.1.3
Facility Life
Cycle
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(LFO)
Remove reference to MREFC panel from all but
design phases
Addressed in this version. Figure
2.1.3-1 was revised, deleted MREFC
from the Renewal & Recompetition
stage.
C-16
2.1.3-1,
last
paragraph
NSF
(GEO)
Figure 2.1.3-1 Progressive Steps
in the MREFC Life Cycle,
…… Renewal & Re-competition
Reviews(Program, MREFC Panel
& DRB, OD & NSB)
“Individuals should discuss any
proposed departures with the
cognizant Program Officer. “
Addressed in this version.
Changed “Individuals” to “Project
stakeholders”.
C-17
2.1.3-1,
last
paragraph
NSF
(GEO)
Who are the individuals in the attached
statement?
Note: If this is an internal document, are
individuals from the Awardee permitted to
provide input?
IMP should be defined prior to this section as the
reader does not have context to the paragraph
meaning. ( It is defined in the later sections of
the document )
C-18
2.1.3-1
Lockheed
Martin
(AIMS)
C-15
Points at which there may be
departure from the MREFC
process outlined here should be
identified early in the project
development and documented as
part of the NSF Internal
Management Plan (IMP)
Individuals should discuss any
proposed departures with the
cognizant Program Officer.
Will there be any documentation that the
Operations and Divestment stage are not part of
this project? The document says any departure
from these stages would be identified in the NSF
IMP but I wasn’t sure if that information was
reflected.
Addressed in this version. Added:
“The IMP is the primary document
that describes how NSF will oversee a
facility through the various life cycle
stages (see Section 3.3).”
Typo addressed in this version. A
period was added after “(IMP)”.
No change. Response to Commenter
was provided:
Per section 7.5 of the PEP, the
Concept of Operation Plan should
include estimate of annual operations
and maintenance staffing and funding
that will be needed when the facility
is constructed and operated.
Per section 7.6 of the PEP, the Facility
Divestment Plan should include a
description and estimate of
divestment liabilities at the end of
facility life for transfer, demolition,
site remediation, decontamination,
Content & Clarify
Page 7 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
C-19
2.1.3-3
Lockheed
Martin
(AIMS)
C-20
2.1.4
Figure 2.1.41
2.1.4
Figure 2.1.41
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
C-21
Content & Clarify
LFM Language/ Observations
Preliminary Design Phase
Definition: Further advances the
project baseline definition and
the Project Execution Plan. It
produces a bottom-up scope,
cost, schedule, and risk analysis
of sufficient maturity to allow
determination of the Project
Total Cost and overall duration
for a given Fiscal Year start and to
establish the MREFC budget
request to congress.
Comment
It is understood we provide a funding profile and
spend plan for PDR. We also understand we
baseline at PDR per the LFM for scope, cost and
schedule. Does the LFO allow for changes in the
funding and spend plan profiles at the end of FDR
to allow for any updates we may need to make
based on the awarded subcontract?
NSF Response
etc. This plan will not be anywhere
near as developed and detailed as the
PEP, but should discuss above items
at a high level.
Addressed in this version. Language
revised in Sections 2.3 and 2.4 to
clarify TPC, baseline, and performance
measurement baseline (PMB).
What is the systems engineering requirement for
Large Facility Projects?
Will be assessed and addressed in
later version.
What is the output or definition of a developed
Project Controls System?
Addressed in this version. Added to
footnote on 2.3.2.-2: “See Figure
4.2.2-1 for examples of project
controls systems inputs and outputs.”
Section 10 of the PEP addresses
Project Management Controls
including EVMS. LFM Section 4.5.3.6 is
reserved for future content on EVM
validation.
Page 8 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-22
C-23
C-24
C-25
C-26
LFM Section/
Paragraph
2.1.4
Summary of
MREFC
Source
2.1.4
Figure 2.1.41
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
OCE)
Can industry lead facility projects or only nonprofits/academic institutions? (This is usually an
FAQ, so perhaps addressing it upfront in the
summary section would be beneficial)
NSF
(GEO)
NEPA and permitting responsibilities (not just
environmental impacts) should be articulated in
the chart. Does one need the EA or EIS prior to
construction funding request?
Addressed in this version. Paragraphs
2-4 in section 2.1.4 were modified to
clarify the fact that various
institutions (educational, non-profit,
or industry) may propose and be
awarded implementation of work for
different stages based on best
interests of a particular project and
NSF goal of nurturing institutions
doing cutting edge science.
Addressed in this version. Figure 2.1.4
was revised to include NEPA.
Additional guidance on NEPA
requirements will be added to the
LFM after internal guidelines are
finalized.
2.1.4
Figure 2.1.41
2.1.4
Figure 2.1.41
NSF
(GEO)
Editorial Note – label Figures on bottom, Tables
on top
What does industrialize key technologies mean?
(not a modern term)
NSF
(GEO)
When is the ConOps (Operations Plan?) delivered
and reviewed?
2.1.4
Figure 2.1.41
NSF
(GEO)
Property management/ownership model should
be included or in the PEP. This is very important
for developing the design. (lease, NSF own,
Awardee own, there are a lot of options to trade
off and assess)
Content & Clarify
Will be assessed and addressed in
later version.
Addressed in this version. Figure 2.1.4
was revised to clarify.
Addressed in this version. Revisions
to define the Concept of Operations
during the planning and the
Operation Plans for Ops.
Will be assessed and addressed in
later version as part of the PEP
detailed guidelines, Section 3.4.2.
Page 9 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.1.4
Fig. 2.1.4-2
Source
C-28
2.1.5
C-29
2.1.6-14
NSF
(GEO/
OCE)
Lockheed
Martin
(AIMS)
C-30
2.1.6-6
Lockheed
Martin
(AIMS)
C-31
2.1.6-7
Lockheed
Martin
(AIMS)
C-27
Content & Clarify
LFM Language/ Observations
NSF
(OIG)
Pg 2.1.6-14. “The CO holds the
warrant and is the only individual
authorized to obligate or deobligate government funds. The
CO, through their warrant, has
the sole authority to award and
administer the construction
contract(s) used in support of
Large Facility projects.”
Table – PD Phase
Table – Construction Stage
Comment
NSF Response
We suggest updating figure 2.1.4-1 to incorporate
new requirements, such as the independent cost
estimate, incurred cost audits, incurred cost
submissions, etc.
This to be written Section 2.1.5 should include
internal competition planning text as part of this
section.
This statement says the “CO has the sole authority
to award and administration the construction
contract (s) used in support of Large Facility
projects. This does not make sense for our
situation – can NSF clarify the CO’s role in our
case.
Addressed in this version. Figure
4.2.1-1 revised to include additional
input to the costs analysis and Section
4.5.3 revised.
Will be assessed and addressed in
later version.
What do they mean by the CO “creates
solicitations for enabling research, workshops,
summer studies…. (the first bullet).
Addressed in this version. Language
revised in Table to indicate the G/AO
and CO supports proposal
development (shared responsibility
with PO).
No change. Response discussed with
commenter. For contracts, the PO
does not have single authority to
approve baseline changes. PO will be
involved in approval per Change
Control Plan. CO will provide
formal/final approval of any contract
changes, per bullets under G/AO or
CO.
Does the PO have authority to approve significant
change to the project baseline?
Addressed in this version. Duplicate of
C-33. Statement clarified/updated to
“The CO, through their warrant, has
the sole authority to award and
administer the prime construction
contract(s) used in support of Large
Facility projects.”
Page 10 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
Comment
NSF Response
Who are the external ad hoc panel members?
No change. See Sections 2.4.1 and
4.5.3.2, for annual review panel
during construction.
Need to understand the “periodic reviews” of
project progress – monthly, quarterly, annually so
we can determine frequency and requirements
for these reviews?
No change. See Sections 2.4.1 and
4.5.3.2 for annual review panel during
construction.
Is the GPRA where the requirement for an annual
performance plan comes from?
C-32
2.1.6-13
Content & Clarify
Lockheed
Martin
(AIMS)
The G/AO is responsible for
oversight of the financial and
administrative terms and
conditions of the assistance
agreement, just as the PO is
responsible for scientific and
technical oversight. Unlike the
PO, he/she holds the warrant to
obligate government funds. The
G/AO and the PO jointly share
the principal technical and
financial responsibilities for the
oversight and assurance of a
large facility project.
Yes, since replaced by Government
Performance and Results
Modernization Act of 2010
(GPRAMA). Updated LFM reference to
state: “ensures compliance with
Government Performance and Results
Modernization Act of 2010
(GPRAMA)”.
What is the difference between the G/AO and the No change. The LFM describes the
CO?
difference between G/AO and CO in
the current version. Response was
made directly to commenter
explaining that the G/AO is the
financial officer for cooperative
agreements, CO for contracts.
Page 11 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
LFM Language/ Observations
Comment
NSF Response
C-33
LFM Section/
Paragraph
2.1.6-14
Lockheed
Martin
(AIMS)
This statement says the “CO has the sole
authority to award and administration the
construction contract (s) used in support of Large
Facility projects. This does not make sense for
our situation – can NSF clarify the CO’s role in our
case.
Addressed in this version. Duplicate of
C-29. Language clarified/updated to
“The CO, through their warrant, has
the sole authority to award and
administer the prime construction
contract(s) used in support of Large
Facility projects.”
C-34
2.1.6-14
Lockheed
Martin
(AIMS)
The CO holds the warrant and is
the only individual authorized to
obligate or de-obligate
government funds. The CO,
through their warrant, has the
sole authority to award and
administer the construction
contract(s) used in support of
Large Facility projects.
Cost Analyst
Who is this on NSF side? Need to confirm that
this is not a cost item that NSF must bear and
verify if this person is in place right now.
C-35
2.1.6-14
Lockheed
Martin
(AIMS)
Footnote 1: Refer to the Business
Systems Review (BSR) Guide
described in Section 4.5.3.3 for
discussion on this point. When
NSF is not the cognizant audit
agency for the recipient
institution, its oversight of
recipient business practices is
narrowly defined.
Will there be another BSR on ASC as there was
already one conducted when the current contract
was awarded? We need to understand timing as
well for when this will take place.
Addressed in this version.
Revised language to “The Cost Analyst
provides a written recommendation
to the G/AO or CO stating… identified
by the requesting G/AO or CO.”
Will be assessed and addressed in
later version.
Content & Clarify
Page 12 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-36
C-37
LFM Section/
Paragraph
2.1.6-15
HFLO Section
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
OCE)
“Prior to NSF requesting NSB
approval to include a proposed
project in a future budget
request, the HLFO contributes to
agency assurance that the project
plans are construction ready, and
that the construction and
operations budgets are
satisfactorily justified. This
assurance comes through
assignment of the LFO Liaison to
the IPT and membership (as
assigned) on various governance
bodies such as the Director’s
Review Board and MREFC Panel.”
I think the LFM language requires definition of
the deliverable that describes the assurance or
certification of the total project cost of an MREFC
or Operations Project. The language is unclear.
Is it a letter to the CFO certifying the TPC? What
gets delivered? I think this is the most important
authority that HFLO has in the document. Who is
responsible for the project independent cost
estimate? Is it LFO or the Directorate?
No change. Comment addresses
internal procedures. Internal
processes are handled with internal
Standard Operating Guidelines
(SOG’s).
2.1.6
HFLO Section
– monthly
report
NSF
(GEO/
OCE)
What are the specific monthly report
requirements for the PO? These requirements
have changed over time, but are not specified in
the manual. In the past, the requirement was to
submit the awardee’s submission that met LFO
requirements with a summary from the PO. This
was not uniformly applied as some POs wrote
their own reports based on the Awardee
submissions. The report sections for an MREFC
project would benefit from standardization as the
Awardees could then use the Manual for
direction. Some CAs have clauses written to
specify reports, so maybe the Manual should
direct the Awardee to the award clause.
No change. NSF internal process. SOG
under development.
Content & Clarify
Page 13 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.1.6
HLFO Section
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
OCE)
The statement “The HLFO
develops and implements
processes for insuring that all
facility award instruments” is
unclear.
What are the processes, how are they
implemented, who delivers and when? Is it a
report, email, discussion? This is unclear.
Addressed in this version. Language
revised in Section 2.1.6 to clarify role
and responsibility of HLFO and LFO.
C-39
2.1.6
NSF
(GEO/
OCE)
Will be assessed and addressed in
later version.
C-40
2.1.6
Roles and
Responsibiliti
es
NSF
(GEO/
PLR)
The roles and responsibilities section contains a
lot of verbs. In many cases the deliverables and
sign off associated with these verbs are unclear.
For example, what does assurance deliver and
how is it delivered? (a report, a letter, email,
witness, controlled document, etc.) This section
would be a lot stronger if the deliverables and
location of the filing of these deliverables would
be defined. Appendix B only has some of the
information.
Include “or contract agreement” where CA
appears.
For third bullet, LFO role does not apply to
Antarctica.
Addressed in this version. Language
was revised from “project
management issues related to
recompetition of award for facility
operation” to “related project
management issues in the event of
recompetition of the award for
facility operations”
C-38
Content & Clarify
Table 2.1.6-1 Summary of
Principal Roles and
Responsibilities:
Under ‘Construction/
Implementation Stage’
“Develops a Cooperative
Agreement (CA)”
Under ‘Operations Stage’
(G/AO or CO)Creates special
terms and conditions in the CA
to capture requirements for
annual performance goals
(shared responsibility with the
PO)
(LFO) Advises PO and G/AO on
project management issues
related to recompetition of
award for facility operation
Addressed in this version. Language in
table was modified to include
contract agreements in addition to
cooperative agreements (CAs).
Page 14 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.1.6
Roles and
Responsibilities
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
PLR)
Not straightforward for PLR; This can be
delegated and often is in that agreements are
signed at the level where resolution of issues is
handled;
Addressed in this version. Language
added “Unless delegated to a lower
level, …” and revised to “…signs or
delegates signature for these…”
C-42
2.1.6
Roles and
Responsibilities
NSF
(GEO/
PLR)
Page 2.1.6-11 “The AD oversees
development of MOUs with
other agencies, international
partners, private foundations,
and other entities and, with the
approval of the NSF OD, enters
into negotiations with those
parties and signs these
agreements on behalf of NSF
when authority to do so is
delegated by NSF OD.”
Page 2.1.6-11 “Overseeing the
organization of all design
reviews including appointment
of review panels, charges to the
panels, and Directorate
responses to review panel
recommendations”
Did this happen before? The DD is meant to be
cut out of the loop here?
C-43
2.1.6
Roles and
Responsibilities
NSF
(GEO/
PLR)
Page 2.1.6-11 “Ensuring that the
performance plans of the
relevant Division Directors reflect
the requirements and
expectations of the LFM and
other NSF policy statements, and
the necessity to provide an
environment of open
communication and transparency
in the management of MREFC
projects”
suspect that if this occurs it is only in a generic
sense
Addressed in this version. Language
revised in Section 2.1.6.2 to clarify
Senior Management of the
Originating Division, Directorate, or
Office Assistant Director or Office
Head responsibilities. There is a
separate section for Division
Director’s role.
Addressed in this version. Language
revised in bullet to clarify.
C-41
Content & Clarify
Page 15 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.1.6
Roles and
Responsibilities
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
PLR)
Page 2.1.6-12
“Grants and Agreements Officer”
It should be explicitly stated up front here that
the IPT will have either a G/AO or CO depending
upon whether a CA or contract mechanism is in
play.
C-45
2.1.6
NSF
(MPS)
Addressed in this revision. The
assignment of a G/AO or CO to the IPT
for CAs or contracts is explicitly listed
in the roles and responsibilities for
those two individuals. Figure 2.1.6-4
lists G/AO or CO as part of the IPT
Award Management Group.
Addressed in this version.
“Experienced and trained in
management of large projects.” was
added to the LFO liaison role in Table
2.1.6-1
C-46
2.1.6, 3.1,
3.3, 4.2, and
4.5
NSF
(DACS)
C-44
Content & Clarify
These sections should be using
the word “closeout” rather than
“termination”
Table 2.1.6-1 says the PO is “Experienced or
trained in management of large projects.” It also
says the LFO Liaison “Advises POs on project
management issues during project development
and oversight” yet there is no requirement that
the LFO have similar experience or training.
2.1.6-5
Table 2.1.6-1
Devises and carries out a renewal or termination
strategy that implements recompetition of the
operating award wherever feasible
2.1.6-7
Devises and carries out a renewal or termination
strategy that implements recompetition of the
operating award wherever feasible
Defines business practices for renewal,
recompetition, or termination of the Award
Advises PO and G/AO on effective operational
oversight strategies, renewal and recompetition
strategies, and termination
2.1.6-10
Overseeing implementation, operation, and
eventual divestment and termination of NSF
support for the project.
2.1.6-13
Addressed in this version. Related to
Comment C-2. The use of the terms
‘termination’ and ‘closeout’ were
revised to conform to the 2 CFR
language for award lifecycle and FAR
definitions. Term definitions were
added to the lexicon.
Page 16 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
Comment
The G/AO confers with the PO and other relevant
offices to ensure that the NSF’s technical and
administrative oversight activities are well
coordinated. The G/AO and the PO collaborate on
the preparation of solicitations and the proposal
and award process. The G/AO has individual
responsibility for developing and overseeing the
implementation of financial and administrative
aspects of the award process, and joint
responsibility with the PO for recompetition
planning and execution, and award termination
and closeout.
2.3.1-3 facility. It also identifies critical issues and
risks facing the project (for example: project
management issues, completing essential R&D
activities, partnership agreements, termination or
divestment liabilities) and lays out a strategy for
financing these activities.
3.1-1
Guidelines for plans to terminate operations
under NSF awards are in development, with
Section 3.6 provided as a placeholder.
Divestment, and termination of NSF funding and
oversight of a facility, may be accomplished
through transfer to another agency or funding
source or through decommissioning and
deconstruction.
3.3-1
This document provides guidance to the PO on
topics to be included in an Internal Management
Plan (IMP), grouped by life-cycle stage. The IMP is
the primary document that describes how NSF
will oversee development, construction,
operation and eventually divestment and
Content & Clarify
NSF Response
Addressed in this version. Related to
Comment C-2. The use of the terms
‘termination’ and ‘closeout’ were
revised to conform to the 2 CFR
language for award lifecycle and FAR
definitions. Term definitions were
added to the lexicon.
Page 17 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
Comment
termination of the award support for a major
facility. The requirement to develop an IMP is
described in Section 2.3.1 for MREFC and in
Section 2.7 for non-MREFC projects. Two primary
purposes are served by
development of an IMP:
4.2.2-1
The CEP should be tailored to address all relevant
stages and costs of the facility lifecycle, from
Development and Design through Construction,
Operation, and Termination /Disinvestment. For
example, the expected level of funding needed
for the Operations Stage should be identified at
the Conceptual Design Review. Operating costs
estimates will be updated throughout the design
and construction process as further discussed in
the Concept of Operations Plan developed as part
of the PEP described in Section 3.4 of this
manual.
4.2.2-2
For most large facilities projects, funding is
derived from the appropriate NSF budget account
depending on the Stage: typically the Research
and Related Activities (R&RA), Education and
Human Resources (EHR) or MREFC account.
Barring documented exceptions, the R&RA (and
possibly EHR) account will be used to fund the
Development, Design, Operations, and
Termination Disinvestment Stage costs. The
MREFC account will be used to fund construction,
acquisition and commissioning costs as part of
the Construction Stage.
4.2.4-1
Content & Clarify
NSF Response
Addressed in this version. Related to
Comment C-2. The use of the terms
‘termination’ and ‘closeout’ were
revised to conform to the 2 CFR
language for award lifecycle and FAR
definitions. Term definitions were
added to the lexicon.
Page 18 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-47
LFM Section/
Paragraph
2.1.6.1 Integrated
Project
Team (IPT)
Content & Clarify
Source
NSF
(OISE)
LFM Language/ Observations
Comment
If anticipated by NSF and as discussed in Section
2.6 of this manual, proposals will be requested to
address partial or full termination disinvestment
of the facility during the award period, including
property decommissioning and disposition costs
and other costs related to employee separations.
It is incumbent upon the Recipient to ensure their
operations proposal is complete, appropriate,
and reasonable.
4.5.1-1
The Recipient is responsible for complying with
the reporting requirements contained in the
award
instrument (e.g., technical and financial
reporting), this manual, and in the Proposal and
Award Policy and Procedures Guide (PAPPG) –
particularly with respect to property
management and final reporting and closeout
requirements for termination of the award. The
Recipient is also responsible for providing internal
oversight of its own activities.
Regarding specific LF projects, please consider
adding an OISE program officer to each LF
Integrated Project Team (IPT). This OISE PO
could provide country-specific guidance on
international partnership for a particular
MREFC project, and as necessary, facilitate
coordination with international partners during
the design-to-execution phases of the project.
If such a role for OISE is agreeable to LFO, then,
a suggested edit to the LFM is:
Sec.2.1.6.2 - Integrated Project Team (IPT) “Comprised of NSF personnel with knowledge
NSF Response
Addressed in this version. Related to
Comment C-2. The use of the terms
‘termination’ and ‘closeout’ were
revised to conform to the 2 CFR
language for award lifecycle and FAR
definitions. Term definitions were
added to the lexicon.
Addressed in this version. Description
of IPT revised to align with the SOG.
OISE is included as part of the
strategic membership.
Page 19 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
C-48
2.1.6.3
IPT
C-49
2.1.6.3
Coordinating
Advisory
Bodies
C-50
2.1.6.4
MREFC Panel
& DRB
Content & Clarify
Source
NSF
(GEO/
OCE)
NSF
(GEO/
PLR)
NSF
(GEO/
OCE)
LFM Language/ Observations
Page 2.1.6-17 “The IPT is
chaired by the PO. See Figure
2.1.6-4. Members are
appointed by the ADs or Office
Heads, in consultation with the
PO. Appointments shall be for
the duration of the project or
until new appointments are
made by the Office Heads. The
PO will convene the IPT at least
quarterly to discuss any projectrelated issues.”
Comment
NSF Response
and expertise in areas related to the scientific
and technical, award management,
international partnership, and strategic aspects
of a particular MREFC project.”
Minutes of the meeting should be required and
communicated to assurance functions.
No change. Addressed in internal
guidelines (SOG).
bypasses DD who is held accountable and
supposed to facilitate communications...
Addressed in this version. Language
revised to include DD’s and AD’s.
correction to “until new appointments are
made by the ADs or Office Heads
What are the deliverables from the MREFC panel
with respect to assurance and prioritization? Is
the facilities plan or budget request timeline
prioritized by the MREFC panel? Is there a
charter for the MREFC panel that describes
functioning and quorum voting, etc.? I think the
information in Appendix B may not suffice.
No change. This is an internal NSF
process and codified in internal
guidelines (SOG).
Page 20 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.1.6-2
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OISE)
Membership of LFWG
No change to LFM. Agreed. LFWG
modified to include OISE as part of
the strategic membership.
C-52
2.2.2
Approvals
NSF
(GEO/
OCE)
C-53
2.2.2
NSF
(OGC)
C-54
2.3.1.1
NSF
(GEO/
OCE)
We also request nominating to the overall Large
Facilities Working Group the OISE program officer
who is assigned to be OISE liaison to LFO. This is
distinct from the OISE participation on specific
IPTs (above). As part of the LFWG advisory group,
the OISE PO could provide big-picture input
regarding international cooperation, US foreign
policy, etc., related to NSF Large Facilities.
What is the format for the approvals noted in this
section? (letter, email, as described by another
policy, etc.) Appendix B has some information,
but not all.
The attached bullet describes exiting from
development to design stage. The “planning”
section states that the Director should be
notified of any COI. My question is what sort of
COI is this referring to? Her COI with the
project if any? The COI of another NSF staff
member?
This section should include a concept of
operations. The basics of how the facility is
envisioned to be operated and the user
requirements as well as the system requirements.
This is more specific and a necessary addition as
it serves to start user/operator requirements that
could be different than a “community outreach”
section of the PEP. The User requirements are
often omitted from the system requirements or
science objectives and missed. This is especially
important for cyberinfrastructure projects. (basic
systems engineering – above)
C-51
Content & Clarify
• Are there any conflicts of
interest or other major
challenges regarding this
project that the Director needs
to be aware of?
Addressed in this version. Language
revised for written approvals.
Addressed in this version. Language
revised. Intent is project challenges
other than technical.
Addressed in this version. Added
bullet for concept of operations.
As noted in response to Comment
number C-62, comments associated
with system engineering will be
assessed and addressed in a later
version.
Page 21 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-55
LFM Section/
Paragraph
2.3.2
C-56
2.3.2
C-57
2.3.2
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
C-58
2.3.2
NSF
(GEO/
OCE)
What are the requirements for operations budget
reviews during the construction phase? How
does NSF assure that cost growth is managed
along with scope and TPC?
C-59
2.3.2 PDR
phase
NSF
(GEO/
OCE)
When do the operations costs or ceilings get
defined? How are they developed and iterated?
C-60
2.3.2.5
NSF
Director’s
Recommend
ation
NSF
(GEO/
PLR)
Content & Clarify
LFM Language/ Observations
NSF
(GEO/
OCE)
Bullet 5: “The NSF Director is
satisfied that external
participation in all phases of the
project (other agencies,
international and/or private
sector entities, etc.) is well
planned.”
Comment
NSF Response
How does the no cost overrun policy apply to
operational costs? If the ops budget changes,
what is the process to right size the scope to
assure that the facility being built can be
operated to cost? This is a lessons learned from
past projects. How is an operational budget
change communicated to the MREFC panel &
DRB post-approval?
The design process does not include text on
design/build vs design/bid/build and how this is
handled with respect to project readiness.
NEPA -- Is the FNSI needed or just
commencement of the process? In some cases
permits involve a lot of risk. How should this be
addressed in the text?
No change. Policy does not apply to
operations. Operations equals
proposed budget, not a total project
cost (TPC).
This is why Rick Spinrad and others went to
Antarctica.
Will be assessed and addressed in
later version after SOG is complete.
No change to the LFM. FNSI is a result
of an EA and not applicable to all
projects. Section 5.5 and Fig 2.1.4-1
indicate that environmental impacts
need to be understood by PDR.
Addressed in this version. Discussion
on the Concept of Operations Plan
was added including the requirement
for an estimate of annual operations
and maintenance costs, staffing
levels, and other activities.
No change required. Figure 2.1.4-1
identifies it is iterated at each stage.
Internal process that needs
assessment.
No change. Comment is an
observation statement.
Page 22 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-61
LFM Section/
Paragraph
2.3.3
Source
LFM Language/ Observations
Comment
NSF Response
Lockheed
Martin
(AIMS)
Page 2.3.3-1 4th bullet and 2.3.3-2
footnote comment: A fully
implemented PMCS, including a
final version of the resourceloaded schedule and
mechanisms for the project to
generate reports – using the
Earned Value Management
System (EVMS)1 – on a monthly
Will this be a requirement for LM which we will
then flow down to our subcontractors?
Addressed in this revision. Footnote
was expanded to clarify. “1 During
construction, progress should be
tracked and measured using the
Earned Value method (this method is
required by OMB in Planning,
Budgeting, Acquisition, and
Management of Capital Assets, OMB
Circular No. A–11 (2014). This is a
requirement for the award recipient
who acts as a prime contractor.
Secondary contractors to the prime
are not required to follow formal
EVMS, but must be able to provide the
appropriate inputs to the EV
reporting. A discussion of Earned
Value is included section of Section
5.8, Guidelines for Earned Value
Management, currently under
development.”
The Manual really needs a systems engineering
description of this bullet and the relationship to
Conops, requirements, derivation, acceptance
(test, integration, verification, validation,
commissioning and Operational Definition. Here
is an example of what is needed:
Will be assessed and addressed in
later version.
basis and use them as a
management tool. Path
dependencies, schedule float, and
critical path are defined; and
footnote 1 During construction,
progress should be tracked and
measured using the Earned Value
method (this method is required
by OMB in Planning, Budgeting,
Acquisition, and Management of
Capital Assets, OMB Circular No.
A–11 (2014). A discussion of
Earned Value is included section
of Section 8, Guidelines for
Earned Value Management.
C-62
2.3.3.1
Bullet 2,
requirements
Content & Clarify
NSF
(GEO/
OCE)
Page 23 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
Comment
NSF Response
Will be assessed and addressed in
later version.
The manual should utilize some sort of INCOSE
process and break down the process to address
the development, construction and transition
processes more clearly than “functional
requirements”.
Given the unique nature of NSF projects, I do
agree that in the conceptual stage there needs to
be traceability from the science question(s) or
science to the infrastructure conceptual design.
In some cases, I think NSF should think about
getting external technical review on the
traceability before starting the investment in
projectizing as this is very expensive and can
cause a project to develop without the proper
scientific input and prioritization. I would also
recommend that projects get a technology
readiness level assessment at the traceability
(pre-concept design review stage) and get
Content & Clarify
Page 24 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
C-63
2.3.3.2
NSF
(GEO/
OCE)
C-64
2.3.3.2
FDR
NSF
(GEO/
OCE)
C-65
2.3.3.3
NSF
(GEO/
OCE)
Content & Clarify
LFM Language/ Observations
Comment
scientific input/external review at that point as
well. The definitions of “industrialization, nonrecurring engineering, subsystem development
and their relation to technology and engineering
readiness need to be included in the Large
Facilities Manual more explicitly.
I recommend adaptation of INCOSE processes
into facility guidance and the Manual. This is a
lesson learned at NSF that needs to be
incorporated into the Manual from multiple
projects.
“Fit up” is not a modern term.
Use an INCOSE lexicon for awardees (interfaces,
ICDs, ICAs, etc.). This is a lessons learned. Again,
the terminology with respect to systems
engineering, requirements, IVT and
commissioning is not modern or to generally
accepted engineering practices. This is a lesson
learned and has been an FAQ at many design
reviews.
Quality and Safety Management should be clearly
articulated at FDR and integrated into schedule
and cost baselines. Change the wording of
“plans” to management and staffing policies,
procedures and staff. This is a lesson learned.
Awardees are responsible for conducting cost
analysis and delivering it to NSF for review. At
what step in the process is the cost analysis from
the Awardee delivered and reviewed? It should
be delivered with every cost estimate at every
design review. Where is the cost analysis
standard for the Awardee provided in the
Manual?
NSF Response
Will be assessed and addressed in
later version.
Will be assessed and addressed in
later version.
Addressed in this version. Added
“Management; integrated into the
resource loaded schedule”.
No change. Response made directly to
Commenter. Cost analysis is done by
NSF and explained with new 4.2
section.
Page 25 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-66
LFM Section/
Paragraph
2.4.1
C-67
2.4.1
NSF
(GEO/
OCE)
C-68
2.4.2
Content & Clarify
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
LFM Language/ Observations
Comment
NSF Response
I think there should be more language to describe
the baseline maintenance process we expect
during construction. The re-plan definition seems
to allow Awardees to allocate schedule
contingency without tracking variances to original
plans. This is also an EVM comment. I think
more guidance is needed on this in the Manual.
Is the OMB requirement to report any impact to a
science requirement in the monthly report still
valid?
How is scope management (descoping) included
in the baseline maintenance section?
Addressed in this revision. Language
revised throughout Section 2.4.1 to
clarify the similarity between replanning and use of cost and schedule
contingency. A sentence was added to
indicate the use of EVMS to track
progress and variances.
Will be assessed and addressed in
later version.
The Commissioning section does not use modern
engineering language. I recommend
incorporating INCOSE language and aligning the
commissioning strategy with the acquisition
strategy. The commissioning plan should be as
detailed as the Final Design and acquisition
strategy allows. If the plan is allowed to float to
year 5 of a 6 year project, it will not have the
proper upfront time and budget allocation.
Lesson learned on many NSF projects. Too often
weak definition of upfront IVT/commissioning
results in transfer of costs and risk to the O&M
phase. The year 1 O&M budget needs to be
clearly reviewed based on the IVT/commissioning
at FDR. Year 1 ops budgets should be higher than
later years as this is a best practice. In some
cases Year 1 should be 20-30% higher than Year
2.
Addressed in this version. Language
throughout section 2.4.1 was revised
to include discussion of scope
management as part of re-planning or
re-baselining.
Will be assessed and addressed in
later version.
Partially addressed in this version.
Additional revisions will be assessed
and addressed in a later version.
References to commissioning plans
were replaced by a suite of
documents listed in the PEP:
Integration and Commissioning,
Acceptance Plans/Operational
Readiness, Transition to Operations,
and Conduct of Operations.
Bulleted list of items to be included in
the 2.4.2 ‘commissioning’ plan was
relevant only to an operations plan
and thus was moved to section 2.5.1
Operations Management and
Oversight.
Page 26 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
LFM Language/ Observations
Comment
NSF Response
C-69
LFM Section/
Paragraph
2.4.2
Commissioni
ng Plan
NSF
(LFO)
In LFM (15-89) Section 2.4.2,
Commissioning Plan, in the
second sentence which reads
“The scope of commissioning
work is to undertake initial
operation of the facility and bring
it up to the design level of
operation in accordance with the
IMP,”
I suggest striking “IMP” and replacing it with
“PEP” or if we have the latitude to revise the
entire sentence, replace it with: “Commissioning
verifies that the substantially complete facility
operates over its full range of capabilities as
specified in final design documents.”
Addressed in this version. Revised
sentence to: “Commissioning verifies
that the substantially complete facility
operates over its full range of
capabilities as specified in the final
design documents.”
C-70
2.4.3
NSF
(GEO/
OCE)
Will be assessed and addressed in
later version.
C-71
2.5.1
NSF
(GEO/
OCE)
Project Close out costs must be part of the TPC.
Therefore this section needs to be completed and
the requirements of this section should be
included as sections in the PDR and FDR sections.
Is project close out an MREFC or R&RA cost? This
interpretation should be in the policy.
There is standard award language that indicates
the Program Officer provides the Awardee with
Annual Work Plan and Annual Report Guidance. I
think these sections should perhaps reference
the Awards as the language provided in these
sections provides a perception that the AWP and
Annual Report are “lightweight” as compared to
the PEP. In some cases, the AWP will have a
framework that is similar to the PEP and
incorporate more user and science engagement.
Content & Clarify
Addressed in this version. Language
revised to clarify the requirements for
AWP and Annual Report. The PEP is
for the construction stage.
Page 27 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
2.5.1
Source
NSF
(GEO/
PLR)
C-74
2.5.1
Operations
Management
and
Oversight
2.5.2
C-75
2.6
C-72
C-73
Content & Clarify
LFM Language/ Observations
Comment
NSF Response
Ops Reviews should review the Operations Plan
or Manual that the MREFC effort produced. It
might also be helpful to review GPRA at the
review and additional metrics for facility
performance that should be established via an
IEEE Conops (if systems engineering language is
incorporated). This is not burdensome, it is
necessary for the management to have a set of
deeper metrics than GPRA to self-assess facility
operations and conduct prudent management.
“Antarctic and Arctic”
Will be assessed and addressed in
later version.
NSF
(GEO/
OCE)
The NSB Recompetition statements and policies
should be cross referenced in this section. The
IMP should be updated at the recompete stage
and included in this section.
NSF
(GEO/
OCE)
Divestment is covered in two places 2.6 and
3.1.1. The discussion on how to manage
divestment prior to defining divestment is
awkward in the Manual, especially since in
section 3.1.1 you state it as undefined. I
recommend in this version that you just the TBD
in 2.6 as well so that the reader does not have to
search ahead for no definition.
Addressed in this version. This
Section and NSB policy is referenced
from Section 3.5.2 Procedures for
Renewal or Recompetition of an
Operating Large Facility with a
footnote. Edits regarding IMP will be
done in later version after SOG IMP
implemented.
Addressed in this version. Language in
Sections 2.6 and 3.1 revised to clarify
the need for divestment plans and to
call out the fact that the section with
guidelines (3.6) is under development.
NSF
(GEO/
OCE)
1st sentence: “Although NSF does
not directly manage the
operations of the facilities it
supports (with the exception of
Antarctic activities),…”
Addressed in this version. “Arctic”
was added.
Page 28 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
Comment
NSF Response
Since the divestment strategy influences
construction strategy, I recommend getting it
defined and reviewed up front. For example
modularization, segmentation and incremental
build might be a necessary construction strategy
based on divestment (LIGO I and LIGO II, etc.).
While meeting with Battelle two weeks ago it
became apparent that brief description in LFM
Section 3.4.1, 10.2 EVMS, may or may not cover a
description what ANSI/EIA 748C describes in
Section 3.7 as the “Earned Value Methodology”
but others may describe as “progressing rules.”
Although one might argue that the word
“processes” covers this, adding “progressing
rules” to the description may better highlight an
area where the sponsor and recipient need to
come to some agreement.
Section 4.4 of the components table "Scope
Management Plan and Scope Contingency"
includes "up-scoping opportunities" remarks. The
intent of this phrase should be clarified, as it is
unclear if this relates to the recovery of prior descopes or allows true scope increases.
Will be assessed and addressed in a
later version (Section 3.6 Guideline
for Divestment plans).
Addressed in this version. PEP
components clarified and Section
3.4.2 reserved for future additional
guidelines for PEP. Section 4.2.2.1
revised to clarify CEP requirements.
Will be assessed and addressed in
later version.
C-76
3.4.1
NSF
(LFO)
10.2 EVMS
C-77
3.4.1 List of
the Essential
Components
of a Project
Execution
Plan, Table-1
AURA
"up-scoping opportunities"
C-78
3.4.1-1
NSF
(GEO/
OCE)
The Project Development Plan elements should
be made clearer as they are foundational to cost
estimation. The CEP development must be in the
PDP
C-79
3.4.1-1
NSF
(GEO/
OCE)
The PDP should include a section on systems
engineering and SEMP
Content & Clarify
Addressed in this version. Language
revised in Section 2.3.3.2.
Addressed in this version. Language
added to clarify in-scope
enhancements and further definition
of the Scope Management Plan in
Section 2.4.1.
Page 29 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-80
LFM Section/
Paragraph
3.4.1-1
C-81
Comment
NSF Response
NSF
(GEO/
OCE)
The PEP or PDP should include the SOP
requirements from the NEPA documentation.
These are sometimes expensive cost drivers
3.4.1-1
NSF
(GEO/
OCE)
The Community Outreach should include the
SOPs created by the environmental documents
and a section on media relations. There are
standard processes used for large construction
efforts across NSF.
C-82
3.4.1-1
NSF
(GEO/
OCE)
C-83
3.4.1-1
NSF
(GEO/
OCE)
Sections 7 and 15 are really the same – see the
systems engineering “V” above. I recommend
leaving Operations in 15 and moving the IVT to
Systems Engineering.
Section 5 – Staffing – some projects may require
an indirect review or Staffing Integration
management review. I think it is important to
highlight the integration between back and front
offices and assure that the proper direct and
indirect support are clearly articulated. This goes
to a description of what each indirect rate applies
at each institution. This will assure that costs
assumed as covered by indirect are actually
covered and that the project, NSF and reviewers
are clear on cost allocations. Sometimes what is
in the rate, does not get applied in practice and
this should be resolved for projects up front.
Addressed in this version. Language
added to Table 3.4.1-1 PEP
components. Revised language in
Section 5.5 and will be further
assessed after NSF internal guidelines
are finalized.
Will be assessed and addressed in
later version. As part of an EA there is
public comment period, need to
review the NSF internal guidance to
determine edits, if any, for PEP
contents or other parts of LFM.
Will be assessed and addressed in
later version.
Content & Clarify
LFM Language/ Observations
Addressed in this version. Language
added to Section 4.2 requiring the
application of indirect cost rates be
articulated in the CEP and BOE.
Page 30 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-84
LFM Section/
Paragraph
3.4.1-1
C-85
3.4.1-1
C-86
3.4.1-1
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
C-87
3.4.1-1
Content & Clarify
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
LFM Language/ Observations
Comment
NSF Response
I recommend that section 13 be named
Environment, Safety and Health (ES&H). NEPA is
a continuous process and needs to be managed
during construction with safety and health.
Usually they are in one scope….after siting….
16.3 – Conops is part of the PDP and systems
engineering effort, not part of project close out.
Addressed in this version. Added
“Environmental” to the PEP-13 in
Table 3.4.1-1.
Divestment is a construction requirement and
needs to be in the PDP/systems engineering
sections (up front, like conops)
The PEP should have a section on deliverables.
This should include: 1. Technical Data Package
(standards, plans, specs, documentation) 2.
Manuals for Operations (Operations Manuals) 3.
Standard Operating Procedures 4. Cybersecurity
and COOP 5. Data specifications and data
lifecycle management specs/procedures.
The deliverable section should also assure the
delivery medium (CDs, paper, website) are
specified. The MREFC and R&RA acquisition
strategy maps to this statement as the TDPs and
Ops Manuals will be different for a turn-key vs
deliver only. This section should form the basis
for funding allocation and auditing.
Addressed in this version. Conops
plan was moved to be under Systems
Engineering (PEP 7.5) table 3.4.1-1.
Addressed in this version. Divestment
Plan was moved to be under Systems
Engineering in the PEP element table
3.4.1-1.
Will be assessed and addressed in
later version.
Page 31 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-88
LFM Section/
Paragraph
3.4.1-1
C-89
3.4.2
NSF
(LFO)
C-90
3.4.2-1
Lockheed
Martin
(AIMS)
Can we get the additional details NSF PO expects
to see in the PEP?
C-91
3.5
NSF
(GEO/
OCE)
Add data management plan in place as an
assessment.
Content & Clarify
NSF
(GEO/
OCE)
LFM Language/ Observations
Comment
NSF Response
Change from divestment to downsizing,
decommissioning and/or divestment. Request
that appropriate high level plans, design
requirements and schedule/cost are attached to
the strategy. What happens if NSF needs to cut
operations costs and downsize during the
construction period? What are the midconstruction off ramps to address this situation?
Section 3.4.2 refers to an internal NSF guidance
document “Guidelines for Development of
Project Execution Plans for Large Facilities.”
Where can the document be found? Is this even
useful and still worth sharing with Programs
internally at NSF? Should this reference be
deleted from the LFM and become a SOG?
Will be assessed and addressed in
later version.
Addressed in this version: deleted
unnecessary references to any
internal NSF guidance. Adjusted the
footnotes and references to internal
documents and/or included direction
to contact PO. Indicated that the
Guidelines for the PEP are in
development in Section 3.4.2. Left
references to specific sections of
PAM, as directions to the PO.
Common in Sec 4.2, among others.
Will be assessed and addressed in
later version. Section 3.4.2 “Detailed
Guidelines for Development of Project
Execution Plans for Large Facilities”
LFM is under development.
Addressed in this version. Added a
bullet for the data management plan
to be in place and ready for
operations.
Page 32 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-92
LFM Section/
Paragraph
3.5
C-93
3.5
C-94
3.5.2
C-95
3.5.2
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
NSF
(GEO/
PLR)
C-96
4.2.1
Content & Clarify
LFM Language/ Observations
NSF
(GEO/
OCE)
Lockheed
Martin
(AIMS)
Page 3.5.2-1 Footnote 1 “Refer
to the footnotes in Section
2.1.6 and 2.7.1 on award
thresholds requiring DRB and
NSB approval.”
Pg 4.2.1-2
Comment
NSF Response
This section is “light” on proposal preparation
requirements. If NSF requires that proposals
meet GAO cost requirements, then this section
needs to encompass the GAO requirements
(WBS, set of requirements, etc., etc.). At first
pass, the Ops Plan would need to have the same
requirements as the PEP for MREFC.
I suggest you change the title of this section to
Management & Operations or Operations &
Management. This is the conventional term.
Does the re-comp guidance map to the NSB
guidance provided in November 2015?
Addressed in this version in Section
4.2.
Did not find footnotes with info in 2.1.6
Addressed in this version. Removed
reference to 2.7.1 and clarified
statement pointing to footnote in
2.1.6.
NSF requires 90-180 days to do a cost analysis
before we progress to the next design phase, yet
the figure 4.2.1-1 reflects this analysis is done
post FDR but prior to construction award. 1 – It
sounds like NSF intends to perform this prior to
FDR, is this accurate and if yes why the deviation
from the LFM.
(PJJ Comment: It is my assumption that in order to
complete a cost analysis post FDR but prior to
award of construction (CLIN 005) we will need to
have our subcontractor proposals received, down
select, site visits, final proposals, orals,
negotiations and LMs analysis. After all of that,
then NSF will start their analysis of the LM
submittal which will take up to 180 days. It will
No change. Response made directly to
Commenter. It is provided as a
notional timeline for planning
purposes. Timeline can and has been
adjusted for projects. NSF and the
Recipient should put together a
timeline that supports all project
needs. NSF typically requires 90 to
180 calendar days to complete a full
review and detailed cost analysis of a
proposal budget prior to proceeding
to the next design phase or prior to
award for operations or construction.
This time will vary depending on
Addressed in this version.
“Maintenance” in title of 3.5.1 was
changed to “Management”.
Will be assessed and addressed in
later version.
Page 33 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
C-97
4.2.1
Lockheed
Martin
(AIMS)
Pg 4.2.1-2
C-98
4.2.1
Cost
Estimating
and Analysis
Overview,
Fig. 4.2.1- 2
NSF
(OIG)
When submitting packages for
cost analysis, Recipients must
submit the following as a
mm1mum:
Cost Estimating Plan per Section
4.2.2.1.
"Cost Model Data Set" per
Section 4.2.2.1.
Reports and Proposals per
Sections 4.2.2.2 and either
4.2.3.2 or 4.2.4.2.
The Work Breakdown Structure
(WBS) and WBS Dictionary per
either Sections 4.2.3.3 or 4.2.4.3.
Content & Clarify
Comment
NSF Response
take LM 30 days to finalize our review and
analysis, NSF 180 days to do the same. Then at
least 30 more days for the NSF to issue the
modification to our prime contract and 20 days
for LM to process the award once funding has
been received – this is 260 days or 8.6
months. This is an extraordinary amount of time
for construction proposal validity period. Three
months is often at the high end of a validity
period range. However, extending this period will
cause subcontractors to price in additional risk
due to cost of materials, resource availability, etc.)
Does NSF expect to do a cost analysis of our
subcontractor proposals (IAW 4.2.1) prior to
award of construction?
project scope, cost, risk, complexity,
and relative importance.
No, NSF does not intend to perform
this prior to FDR.
Please clarify whether the steps apply to facilities
in construction, operations, or both. For
example, it was unclear in the operations section
whether WBS is required for operations awards
or not.
Addressed in this version– Sections
4.2.1 and 4.2.2 are applicable to both
construction and operations. WBS is
for both construction and operations.
Added Section 4.2.2.7 Work
Breakdown Structure and revised
language throughout sections 4.2.1
through 4.2.4 to clarify.
Addressed in this version.
Added bullet - For proposals that
contain a subaward(s), each subaward
must include a separate budget
justification. Response also made
directly to Commenter.
Page 34 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-99
C-100
C-101
C-102
C-103
LFM Section/
Paragraph
4.2.1 Cost
Estimating
and Analysis
Overview
4.2.1 Cost
Estimating
and Analysis
Overview
4.2.2
Source
4.2.2
Cost
Estimating &
Analysis,
4.2.2-10
4.2.2
Cost
Estimating
and Analysis
Process,
4.2.2-10
Content & Clarify
LFM Language/ Observations
NSF
(GEO/
PLR)
Comment
NSF Response
Recommend this section be scrubbed to ensure
inclusion of contract mechanism & CO
involvement
Addressed in this version. Added CO
wherever the G/AO was mentioned
and revised caption for figure 4.2.1-1.
NSF
(GEO/
PLR)
Figure 4.2.1-2
Not apropos to the Antarctic support contract
Addressed in this version. Figure
deleted. New figure to illustrate the
NSF cost analysis process.
Lockheed
Martin
(AIMS)
Pg 4.2.2-8 Management Fees
This needs to be clarified for situations involving
contracts. This addresses cooperative
agreements. Contracts need to be addressed
differently or its specific exclusion noted in the
IMP to avoid confusion (see Pg 2.1.3-1).
Addressed in this version. Language
regarding fees has been changed. See
Section 4.2.2.5.
NSF
(OIG)
As a term and condition of the
award, the recipient will be
required to provide information
(typically annually) on the actual
use(s) of the management fee.
The LFM does not require the
auditee to submit a written
assertion of need that details all
sources of revenue, and examine
all federal and non-federal
sources of revenue for each
awardee in determining whether
a management fee is necessary
and warranted.
We suggest NSF more definitively outline the
frequency of submitting management fee data.
We suggest annually instead of "typically
annually".
Addressed in this version. Language
regarding fee has been changed. See
Section 4.2.2.5.
We suggest NSF require the auditee's proposal
involve a full disclosure of all financial resources
and other sources of income available to cover
unallowable expenses.
Addressed in this version. Language
regarding fee has been changed. See
Section 4.2.2.5.
NSF
(OIG)
Page 35 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-104
C-105
C-106
LFM Section/
Paragraph
4.2.2
Cost
Estimating
and Analysis
Process,
4.2.2-10
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OIG)
The LFM lists appropriate uses of
management fee stating,
"Examples of potential
appropriate needs include ....and
financial incentives to obtain and
retain high caliber staff."
Addressed in this version. Language
regarding fee has been changed. See
Section 4.2.2.5.
4.2.2
Cost
Estimating
and Analysis
Process,
4.2.2-10
4.2.2 Sample
Project
Management
Control
System Flow
Chart, Fig. -1
NSF
(OIG)
The LFM does not address
whether management fee should
be captured under a separate
line item in proposals.
Our review of management fee uses at large
facility awardees highlighted examples of
"incentives," including significant signing bonuses
and payment for private school tuition of
employees' children.
The use of management fees for these types of
expenses may be questionable to taxpayers. We
suggest NSF consider giving examples of
"appropriate" financial incentives.
We suggest NSF require management fees to be
submitted as a separate line item in its future NSF
proposals, rather than including them under a
category of allowable cost.
Regarding Risk Analysis Tools, the flow chart only
shows only one approach; Monte Carlo
Simulations. We recommend that other
approaches be documented (e.g. bottoms up
methodology using Cost, Schedule and Technical
risk factors)
No change. This figure is a sample.
Monte Carlo is considered a “gold
standard”. Other methods usually
does not include all possible
correlations and outcomes for
impacted activities from a single risk.
See Section 5.2.8 for more on
quantitative risk analysis.
Content & Clarify
AURA
Addressed in this version. Language in
Section 4.2.2.4 revised to indicate Fee
to be included under G.6 – Other.
Page 36 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-107
C-108
LFM Section/
Paragraph
4.2.2.1
Cost
Estimating
Plan, 4.2.2-1
and 4.2..3-2
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OIG)
Please clarify whether the GAO Cost Estimating
and Assessment Guide is a requirement for
facilities in construction, operations, or both. On
pg. 3.5.1-1 in the operations plan section it states
"Note that cost estimating should follow the
Government Accountability Office (GAO) Cost
Estimating Guidelines, per Section 4.2."
However, 4.2 is unclear what applies to
construction and what applies to operations.
In addition, while the construction section
requires a statement from the awardee that they
followed the GAO Cost Estimating and
Assessment Guide the same requirement is not
made for operations.
Address in this version. Language in
Section 4.2.2.1 was revised to add
clarity. The GAO guide applies to the
facility lifecycle (construction and
operations) but then addresses
primarily construction.
4.2.2.2
Relevant
Guidance
and
Reporting
Formats
NSF
(OIG)
For construction awards, the CEP
must explain how the Recipient
will follow "The Twelve Steps of a
High-Quality Cost Estimating
Process" from the GAO Cost
Estimating and Assessment
Guide.
Recipients are required to follow
the best practices within the GAO
Cost Estimating and Assessment
Guide and GAO Schedule
Assessment Guide.
This is not called out as a
requirement in the operation's
estimates section of the LFM.
Budget contingency should also
be presented as a part of the
total amount of Other Direct
Costs and included in section G.6
on the standard NSF budget
form. Budget contingency budget
estimates should be developed in
accordance with Sections 4.2.5
and 5.2 of this manual. Budget
contingency and allocations of
contingency will be called out in
the Cooperative Support
Agreement by the GIAO under
the "Contingency" section, based
on information provided in the
negotiated budget justification.
Comment: The new Large Facility Manual
outlines how NSF awardees should develop
contingency in Sections 4.2.5 and 5.2 including
development of a risk management plan, tying
contingency to WBS elements, use of Monte
Carlo simulations etc. We suggest 4.2.2.2 be
clear that the proposal budget should include
adequate documentation to support that the
contingency amounts were developed in
accordance with 4.2.5 and 5.2 and are
supportable.
Addressed in this version. Following
was added to 4.2.2.4 under Other
Costs: “The proposal should include
adequate documentation on the basis
of estimate for the contingency
amounts, indicating that they were
developed in accordance with 4.2.5
and 5.2 and are supportable.”
Content & Clarify
Page 37 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-109
C-110
C-111
LFM Section/
Paragraph
4.2.2.2
Relevant
Guidance
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(MPS)
“Proposed budgets must
comply with the applicable
federal regulations, as
implemented by NSF in the
Proposal and Award Policies
and Procedures Guide
(PAPPG)”. Recipients are
required to follow the best
practices within the GAO Cost
Estimating and Assessment
Guide and GAO Schedule
Assessment Guide. Recipients
must note any departures from
these GAO guides and explain
their rationale.”
Addressed in this version. Language
in Section 4.2.2.2 revised to provide
clarity.
4.2.2.2
Relevant
Guidance &
Reporting
4.2.3.2
Cost report
NSF
(GEO/
PLR)
Section 4.2.2.2 “Relevant Guidance” needs
editing. The first part doesn’t need saying – as
NSF never tells proposers that they do need to
comply with federal regulations. As the title is
“Guidance”, instituting the requirement to
follow best practices restricts flexibility that
can be potentially beneficial in a scientific
research environment. The logic of the
paragraph is further tangled when it concludes
that the recipients apparently are not required
to adhere to best practices but can instead
explain their rational for deviating. (I support
that flexibility – but the logic of the paragraph
is a muddle and needs to be rewritten
coherently.)
Be sure language in section is inclusive of
contract mechanism; comply with FAR...;
cognizant CO....
I recommend having the report signed by the
AOR and PI prior to submission. I also
recommend that the Awardee Cost Analysis of
the construction report be an appendix of the
cost report
Will be assessed and addressed in
later version.
Content & Clarify
NSF
(GEO/
OCE)
Addressed in this version. Language
revised to include CO.
Page 38 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-112
C-113
LFM Section/
Paragraph
4.2.2.3
Funding
Sources and
Award
Instruments
4.2.2.4
Content & Clarify
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
PLR)
“All lifecycle Stages are funded
through the appropriate award
instrument (typically a
Cooperative Agreement (CA))
between NSF and the managing
organizations (i.e., the Recipient).
Infrastructure utilization is
typically supported through
grants (R&RA or EHR) funded by
NSF and/or other agencies to
conduct research and education
activities using the facility.”
If contracts are not referred to, it sets AIMS up
for ongoing confusion
Addressed in this version.
Sentence modified to “All lifecycle
Stages are funded through the
appropriate award instrument (a
Cooperative Agreement (CA) or a
contract agreement) between NSF
and the managing organizations….”
Pg 4.2.2-7 Section 4.2.2.4
G.5: Clarification on the reference to uniform
guidance, 2 CFR section #200.331 is incorrect so
needs to be fixed. GPO.gov says parts 305-399
are reserved.
Would like clarification on intent of this reference
or an alternate source document for its
availability.
https://www.gpo.gov/fdsys/search/pagedetails.a
ction?collectionCode=CFR&searchPath=Title+1%2
FChapter+III%2FPart+304&granuleId=CFR-2014title2-vol1-sec200-331&packageId=CFR-2014title2vol1&oldPath=Title+1%2FChapter+III&fromPageD
etails=true&collapse=true&ycord=479
Lockheed
Martin
(AIMS)
Added statement to be verified and
finished “For projects to be funded
through contractual agreements
rather than assistance awards), the
PO recommends to award or decline
in accordance with the proposal
processing procedures contained in…“
Addressed in this version. Language
was added to indicate “Recipients of
cooperative agreements” follow
uniform guidance, 2 CFR section
#200.331.
Addressed in this version. Following
was added to Section 4.2: “For
proposals that contain sub-awards,
each sub-award must include a
separate budget justification.”
Page 39 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
4.2.2.4
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
PLR)
B.3 – Graduate Students
“Recipients should provide …..
Actual payroll data may not be
available as these may be TBH
positions.”
Define TBH
Addressed in this version. TBH stood
for To-Be-Hired. Language changed
to identify open versus filled
positions.
C-115
4.2.2.4 &
General
NSF
(GEO/
OCE)
Addressed in this version. Language
revised to clarify the LFM
supplements the PAPPG.
C-116
4.2.2.4 Suppl.
Guidance for
Construction
& Operations
Awards:
NSF
(DFM)
4.2.2.4 Supplementary Guidance
for Construction and Operations
Awards:
C-117
4.2.2.4
Supplementa
ry Guidance
for
Construction
and
Operations
Awards
AURA
"sub- awards"
C-118
4.2.2.5
Management
Fee
NSF
(MPS)
What is the hierarchy of compliance for
documentation? Is it the supplemental info in
this Manual or the PAPPG? Shouldn’t the Awards
Specialist be the person who is providing
guidance on this matter (maybe via the Program
Officer)?
Recommend adding specific PAPPG and/or
Uniform Guidance references to the sub
paragraphs. For example 4.2.2.4.ETravel:
could reference the PAPPG section on Travel
(Chapter II.C.2.g.(iv)) and/or perhaps 2 CFR
§200.474.
To preclude confusion and to ensure consistency
with Section 2.3.3.3 language, the word "subawards" should be replaced with "sub-awards
and subcontracts" in both sub-sections G.5 and
G.6 of Section 4.2.2.4. Given that the OMB
Guidance addresses the terms "sub-awards" and
subcontracts" separately, and noting that Section
2.3.3.3 of the Manual also addresses the terms
separately, we suggest that Section 4.2.2.4
should read in the same manner.
There is no discussion of performance incentives
in the discussion of management fee. This has
been used by NSF at times.
C-114
Content & Clarify
Addressed in this version. Added
reference to Chapter II.C.2.g in
Section 4.2.2.4.
Addressed in this version. Sub-awards
was expanded to include subcontracts in Section 4.2.2.4, G5, a
note was added to explain the
difference between sub-awards and
sub-contracts in both this section and
on page 4.4-1, and the two terms
were added to the Lexicon.
Addressed in this version. Language
regarding fee has been changed. See
Section 4.2.2.5.
Page 40 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
4.2.2.-7
Cost
Estimating
and Analysis
Process, G.60ther
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OIG)
Budget contingency should also
be presented as part of the total
amount of Other Direct Costs and
included in section G.6 on the
standard NSF budget form.
Addressed in this version. Section 4.2
supersedes the use of SF 424C.
Language added to Section 4.2.2.4 for
contingency to be itemized separately
under “Other Direct Costs”.
C-120
4.2.2-9
NSF
(MPS)
Prohibited use of management
fees.
C-121
4.2.3
Lockheed
Martin
(AIMS)
Pg 4.2.3-5 Construction Cost Book
Detail and Supplementary
Guidance
The contingency estimates include indirect costs
as well as direct costs, so it is not correct to
categorize contingency as only "Other Direct
Costs." Contingencies are required budget
information for construction awards specifically
listed on Form SF 424C (line 13) which NSF has
agreed to use. We suggest that budget
contingency have its own separate line item on
proposed budgets to NSF.
The NSF large facilities manual seems like a
very odd place to look for this information.
Obviously NSF needs to make these policies
clear to awardees, but it might be better to
direct the reader of the LFM, via a hyperlink or
reference, to a more general source for
information on this topic such as the CFR. That
way the LFM will be current if the CFR is
updated.
Is this how they want to see our subs proposals?
Do subs need to submit BOEs for each labor
category? Do they need to use NSF budget
category codes? If yes, do we want to have the
GC use A-I or just G-5 for all BOEs (this section
refers back to section 4.2.2.4)? While I am not
used to doing this for construction it is not
impossible. It should be noted that in addition to
BOEs there will be need a detailed cost estimate
by WBS in excel to match up with each BOE.
Requiring a General Contractor to provide this
level of detail for each labor category seems
unnecessary. This may be more appropriate for
contracts that are not fixed price.
C-119
Content & Clarify
Addressed in this version. Language
regarding fee has been changed. See
Section 4.2.2.5.
Addressed in this version and direct
with commenter. The information is
intended to supplement the GAO best
practices, grant guidance in the
PAPPG and industry standards.
Language in Sections 4.2.2 and 4.2.3
revised to clarify.
Page 41 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-122
LFM Section/
Paragraph
4.2.3.3
C-123
4.2.3.3
C-124
4.2.3.4
Content & Clarify
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
OCE)
Emphasize that the WBS should be deliverable
based. Emphasize activity based estimates versus
LOE. To me, it looks like you are encouraging all
IVT to be LOE. I think a statement on minimizing
all LOE where practicable would be good
guidance.
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
Define integration, verification, validation and
test. Recommend that this get done at
subsystem, system and system of systems levels.
Is BOE refinement the Awardee Cost Analysis? It
seems confusing.
Addressed in this version. Section
4.2.3.4 indicates that the WBS should
be deliverables based.
Language added to 4.2.3.3: Level of
effort tasks should be minimized for
optimizing tracking of spending
against budget and accomplishments
against plan in the project Earned
Value Management reports.
Will be assessed and addressed in
later version.
No change. Response made to
commenter. Cost analysis done by
NSF.
Page 42 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-125
C-126
LFM Section/
Paragraph
4.2.4
Cost
Estimating
and Analysis
for
Operations
Awards,
4.2.4-1
through
4.2.4-4.
4.2.4-1
figure
Content & Clarify
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OIG)
The language in the construction
estimate section includes:
"The results of an independent
cost estimate review will be
factored into the NSF cost
analysis. To ensure maximum
usefulness of the review, it will
generally be done as part of the
Preliminary Design Phase."
"There may be one or more years
between the PDR and the start of
construction, which
is predicated on successful
completion of the FDR."
The cost estimating and analysis for operations
awards section is silent on whether independent
cost estimate reviews will be done on operations
awards. We suggest NSF require that significant
operation awards go through an independent
cost estimate review. We suggest NSF should
make it clear when the independent cost
estimate will be done so awardees can plan
accordingly. Currently, the LFM says "will
generally be done as part of the Preliminary
Design Phase." If there are significant changes
between the PDR and FDR how will NSF assess
the new quotes and data? We suggest NSF
conduct an independent review of all updated
quotes. If the independent cost estimate review
indicates significant issues, or significant time has
elapsed between the PDR and FDR, how will NSF
handle this? We suggest NSF request the
recipient to update its estimates with current,
accurate and complete cost data and also
perform a follow-up review. We suggest that the
large facility proposals go through the most
stringent of the eight types of GAO independent
cost reviews, an independent cost estimate.
Education & Outreach budgets (EHR vs
directorates). I recommend putting (if allowable)
next to this term
Addressed in this version. Following
language added to section 4.2.4: NSF
may choose to perform independent
cost reviews of operations awards
based on a risk-based determination
of need and review type.
NSF
(GEO/
OCE)
No change needed. Words describe
expectations for more refined
estimates and a NSF cost analysis.
No change. Expect to do ICE, but
don’t need to require it for all.
No change. The figure is a sample.
Page 43 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
4.2.4-1
figure
Source
C-128
4.2.4-1
figure
C-129
4.2.5.1
NSF Policy
Positions,
Point 4
NSF
(GEO/
OCE)
NSF
(OIG)
C-127
Content & Clarify
LFM Language/ Observations
NSF
(GEO/
OCE)
Bullet: 5.0 Contingency
In support of NSF's "No Cost
Overrun" policy, projects shall
use a confidence level for
contingency estimates between
70 and 90 percent (under a
probabilistic approach).
Comment
NSF Response
This figure seems like it is encouraging LOE for
Operations. I think there should be guidance on
encouraging activity based estimates where
practicable. This is a standard M&O panel
recommendation and should be incorporated
into the Manual
Comment addressed in this version –
Changed the list under Science
Operations from
2.0 Principal Investigator(s)
2.1 Scientists
2.2 Postdoctoral Scholars
2.3 Students
to
2.1 Research Planning
2.2 Experimental and
Operations Support
2.3 Data Analysis
2.4 Calibrations and Data
Quality
2.5 Special Projects
Reference the definition of Operations
Contingency in the figure. In the section, hotlink
the Uniform Guidance
We suggest NSF also state its expectation not
only in terms of a confidence level, but also the
precision of the estimate (so that NSF can say, for
example, that it is 80% confident that the point
estimate is accurate within a range of plus or
minus 20%).
Addressed in this version. Section
4.2.6 references the Uniform
Guidance.
No change. The accuracy of the
estimate changes with development
of the design. The risk register
includes uncertainty in the estimate
for development of the budget
contingency.
Page 44 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-130
C-131
LFM Section/
Paragraph
4.2.5.1
NSF Policy
Positions,
Point 5
Source
LFM Language/ Observations
Comment
NSF
(OIG)
We suggest describing the process and
Addressed in this version. Added
procedures related to this new policy of holding
language to Item 3 on page 4.2.5-4.
budget contingency, including explaining how the
amount of contingency to be held by NSF will be
determined. Also, indicate that the allocations
would be included in the award notice and the
CSA amended. Additionally, the new Contingency
SOG (under Monitoring and Reporting of Budget
Contingency) requires that the Project monitor
budget contingency use against both the NSFapproved contingency budget and the total
allocation to date. This expectation should be
made clear in the LFM. We suggest the LFM be
updated to align with the SOG, i.e., incorporate
requirements from the new SOG as it relates to
the recipient.
4.2.4.4
Operations
Awards
Proposals
Supplementa
ry Guidance
NSF
(GEO/
PLR)
NSF will hold budget contingency
through project completion, in an
amount up to 100% of the total
NSF-approved contingency
budget, until it can be justified
for obligation.
However, the LFM does not
outline the procedures related to
this new policy; for example, that
the allocations would be included
in the award notice and the CSA
amended. As another example,
the new SOG states the project
would monitor budget
contingency against both the
NSF-approved contingency
budget and the total allocation to
date. This expectation should be
made clear in the LFM.
“It is the Recipient’s
responsibility to manage and
maintain the NSF-funded
facilities, equipment, and
instrumentation used in the
conduct research. However, NSF
rarely maintains ownership to
major research equipment and
facilities it funds. This
stewardship responsibility is
necessary to protect the U.S.
Government’s and the public’s
investment in these unique
research facilities.
Content & Clarify
With the exception of polar research station
facilities, ....
NSF Response
Addressed in this version.
Footnote added:
“The polar research station facilities,
for example, are exceptions.”
Page 45 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-132
LFM Section/
Paragraph
4.2.5.1
Content & Clarify
Source
LFM Language/ Observations
Comment
NSF Response
AURA
Provision 5 on Section 4.2.5.1
states "NSF will hold budget
contingency through project
completion, in an amount up to
100% of the total NSF-approved
contingency budget, until it can
be justified for obligation."
It should be noted that the NSF has already
approved the contingency budget through the
rigorous exercise noted in Section 4.2.5, so clearly
the contingency amount (in aggregate) has been
justified. In addition, as per 'standard'
Cooperative Support Agreements, the NSF
requires "All proposed changes or proposed use
of contingency will be made known to NSF
(posted via a change control database or similar)
prior to authorization." It would thus appear that
justification before approval is already
established. It is not clear what additional
justification is required by this provision.
Also, it is unclear from this language as to
whether the contingency funds will be
provided as part of the annual funding action
consistent with the funding profiles already
established as part of the NSF process. If it is
not, then there would potentially be delays in
timely receipt of funds for contingency events;
this could increase project costs. This change
could restrict the ability of the projects to
proactively manage risk (since funds may not
be available as planned when needed -this
again would drive project costs).
With regards to determining the amount to
withhold, the burden to predict usage and the
amount of contingency to allocate is a new
administrative task that has the potential to
drive new tools or new features to existing
tools. The new NSF expectation for risk levels
and project health to be verified through
Monte Carlo analysis of integrated cost,
schedule and risk plans is not immediately
Addressed in this version. Added
language to Item #3 on page 4.2.5-4.
For cooperative agreement
contingency will be held by the
Recipient per the Uniform Guidance
as stated on page 4.2.5-2.
Page 46 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
C-133
4.2.5.1
NSF
(MPS)
C-134
4.2.5.4
Contingency
Planning and
Assessment
during
Preliminary
Design
NSF
(OIG)
Content & Clarify
LFM Language/ Observations
Use of rigorous probabilistic cost
estimating methods that
estimate confidence levels for
the TPC (such as Monte Carlo
methods based on probability
distributions for risk) are
preferred and NSF highly
encourages application of these
methods where practical.
Comment
suited to a time phased assessment. If
historical performance and algorithmic models
can be used this burden will be minimized but
this policy is a noticeable increment in periodic
administrative effort.
NSF Policy position is inconsistent in stating
that NSF has a “No Cost Overrun” policy and
that that Directorates are responsible for the
first 10% of the cost overrun. Item 5 says “NSF
will hold budget contingency through project
completion, in an amount up to 100% of the
total NSF-approved contingency budget, until it
can be justified for obligation.” This provision
may skew the budget profile requested by a
project towards higher award amounts in early
years, since the year-by-year funding must
exceed the sum of expenditures, obligations,
and risks in a given year.
A Project Manager, in dealing with the
uncertainty in the level of availability of budget
contingency, may overestimate its potential
need in early years. How will NSF protect
against this?
We suggest NSF make it clear whether
probabilistic cost estimating methods are, or are
not, a requirement. If probabilistic cost
estimating is not used, what are other acceptable
methods?
NSF Response
Addressed in this version. Language
added regarding mechanisms on page
4.2.5-1. Contingency use refers to
budget allocations, not expenditures.
No change. Not required, but industry
standards driven by Section 4.2.5.
Recipient would have to propose
alternate method and NSF would
have to approve.
Page 47 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-135
C-136
LFM Section/
Paragraph
4.2.5.5
Development
of the
Contingency
Use Process
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OIG)
When use of budget contingency is requested,
we suggest that NSF require the recipient to
document in the Change Control Request (in
Additional Documentation for example) which of
its accounts the related charges will be recorded
under to provide better traceability in its
accounting system.
Addressed in this version.
Sample CCB form change from “may”
to “must” for inclusion of cost control
accounts.
4.2.5.6
Contingency
Planning and
Assessment
during Final
Design
NSF
(OIG)
The CCB change request
document, whether forwarded to
NSF for approval or not, must
have the minimum content
requirements necessary to
comply with relevant cost
principles, as well as to maintain
an audit trail. SEE SAMPLE
CHANGE CONTROL REQUEST
FORM. This form states "Cost
control accounts may be included
for traceability in the accounting
system."
At the Final Design Review (FDR)
the budget estimate should be
substantially based on externally
obtained cost estimates (vendor
quotes, bids, historical data, etc.).
This added definition is expected
to result in an increase in the
project's estimated Budget at
Completion
(BAC) and a reduction in its
budget contingency, while TPC
remains constant. Also as part of
the FDR, NSF assesses the
methodology employed by the
project to further refine its cost
and contingency estimates
including schedule and scope
adjustments.
It would be useful if the contingency section
explained whether the independent cost
estimate at PDR would cover a review of
contingency.
Also, if contingency amounts are further refined
between PDR and FDR, we suggest stating that
NSF will review the revised amounts and
specifying what type of review will be performed.
No change. GAO guide explains what
different types of reviews would focus
on. Expect all would review
contingency.
Content & Clarify
No change. This is covered in Item 2
on page 4.2.5-4.
No change, addressed by 5.2.11.1 and
explanation of NSF cost analysis in
sections 4.2.
Page 48 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-137
C-138
C-139
C-140
LFM Section/
Paragraph
4.2.5.6
Contingency
Planning and
Assessment
during Final
Design
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
PLR)
“This added definition is
expected to result in an increase
in the project’s estimated Budget
at Completion (BAC) and a
reduction in its budget
contingency, while TPC remains
constant.”
Why does better information necessarily
translates to an expected increase??
4.2.5.6
Contingency
Planning and
Assessment
during Final
Design
4.2.5.7
Contingency
Use and NSF
Oversight
during
Construction
4.2.5.8
NSF
(GEO/
PLR)
“All of this information would
then factor in to the total project
cost assessment being refined
and evaluated by other divisions
within NSF to make the initial
construction award.”
Unexpended contingency funds
may not be used to support
operations or other out-of-scope
activities.
should clarify what Divisions this is meant to
signal (here and earlier in the document)
Addressed in this version.
Revised to “This added definition is
expected to result in a change in the
project’s estimated Budget at
Completion (BAC) and in the
accompanying budget contingency,
while keeping the sum of the two at
or below the NSB approved TPC.”
Addressed in this version.
Revised to “evaluated by NSF”,
eliminating reference to other
divisions. The process is detailed
elsewhere in this section.
Content & Clarify
NSF
(OIG)
NSF
(GEO/
OCE)
We suggest that NSF clearly define scope and
out- of-scope activities to avoid the possibility of
subsequent disagreement between NSF and the
recipients as to the meaning of these terms.
Addressed in this version. The scope is
articulated in the WBS and scope
management plan.
I think putting the “second level” WBS
requirement may skew the scope delineation. I
recommend that the Awardee propose a
reporting level and the panels/NSF decide on the
proper reporting.
Addressed in this version. “Second
level” revised to “for each element at
the level proposed by the recipient
and agreed upon by NSF”
Page 49 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-141
LFM Section/
Paragraph
4.2.5.8
Source
NSF
(MPS)
C-142
4.2.5.8
Reporting
Requirements
NSF
(OIG)
C-143
4.2.5.9
C-144
4.2.5-1
Change
Control
NSF
(GEO/
OCE)
NSF
(GEO/
OCE)
Content & Clarify
LFM Language/ Observations
At a minimum, the monthly
report will include:..(4) an
updated change log indicating all
contingency allocations ("puts
and takes") and a "liens" list of
projected amounts of possible
future calls on contingency.
Comment
NSF Response
Monthly reporting is a tricky issue – it should
be done monthly but the paper work reduction
act says we cannot compel an awardee to do
so. This section entirely omits reporting
technical status! It’s only financial status! It is
very important that the technical status be
reported, accompanied by a narrative of
progress relative to the plan and the factors
that contribute to cost and schedule variances.
The liens should be listed by date for a decision
on whether to initiate a CCB action. A timephased risk profile is helpful.
We suggest that NSF also request its recipients to
provide a comparison of approved budget to
actual use of contingency. This comparison could
be the recipients' affirmation that they are using
contingency funds as approved by NSF.
No change. Section 4.2.5 covers
Budget Contingency Planning for the
Construction Stage and this
subsection includes the associated
reporting requirements. Section 2.1.6
indicates that the monthly report
summarizes both technical and
financial status.
What about in-kind contributions? How should
these be handled?
Will be assessed and addressed in
later version.
There should be a ‘from/to’ table for rebudgeting and this should be encouraged in the
Project Controls Implementation section (or
designed). The table on the bottom of the
sample form may not allow for clear traceability
across accounts, only drawdown. The process for
indirect allocation and financial officer sign off
(prior to NSF sign off) should be included. This is
a lesson learned. Indirect allocation from
contingency can then get tracked as well (with
proper from/to table)
Addressed in this version. Language in
Section 4.2.5 revised including
clarification of Reporting
Requirements.
No change. Section 5.2.11.3 addresses
Liens List in detail including
anticipated decision date for action.
Addressed in this version. Language
added to Item 3 on page 4.2.5-4 and
reporting requirement on page 4.2.510.
Page 50 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
C-146
LFM Section/
Paragraph
4.2.6
Budget
Contingency
Planning
during the
Operations
Stage
4.3
C-147
4.4
NSF
(GEO/
OCE)
C-148
4.4
Documentation Requirements
NSF
(OIG)
C-145
Content & Clarify
LFM Language/ Observations
Comment
NSF Response
NSF
(MPS)
Needs a better title to avoid the implication that
it is about including budget contingency in
operation. It includes a paragraph about
divestment that would benefit from its own
heading.
NSF
(GEO/
OCE)
Use INCOSE language to describe systems
engineering and a “V” diagram. The SEMP should
have an outline or set of criteria. The acceptance
in the Quality program or subcontracts and
acceptance at the system/subsystem
performance should be clear. (acceptance to
spec, acceptance to performance requirements
with defined levels (user, witness, cert, etc.)
There are many documents that are missing from
this section. Go through the PEP, add the
Operations Manual and develop a list of what is
required to recompete the facility. This will
provide the comprehensive document list NSF
needs. Plans, specs, SOPs, workflows, QA/QC,
safety, cybersecurity, data management and
many more. IF NSF needs to recompete, what is
the acceptable quality level for the
documentation and how do the documents get
delivered?
We suggest adding NSF OIG to the list of
organizations for which the awardee must
provide access to documentation.
Addressed in this version.
Material not relevant to budget
contingency during operations was
moved to section 4.2.4 Cost
Estimating and Analysis for
Operations Awards and 4.2.4.1
Purpose.
Will be assessed and addressed in
later version.
Access to any pertinent books,
documents, papers and records
should be made available to the
NSF Director and the Comptroller
General of the United States or
any of their duly authorized
representatives to make audits…
Will be assessed and addressed in
later version.
Addressed in this version. Section 4.4
revised to include OIG.
Page 51 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
4.5.3
Recipient
Performance
Reviews and
Audits
Source
C-150
4.5.3-3
NSF
(LFO)
C-151
4.5.3
Recipient
Performance
Reviews and
Audits,
4.5.3.4
NSF
(OIG)
C-149
Content & Clarify
LFM Language/ Observations
AURA
NSF conducts a cost incurred
audit for large facility awards
above $1OOM at the end of the
award and potentially during
execution of the award based on
an annual large facility risk
assessment conducted by the
Large Facilities Office and the
Cooperative Support Branch at
NSF.
Comment
NSF Response
The review processes documented in Sections
4.5.3.2 through 4.5.3.4 are in essence "open
ended" in number of reviews that may take
place. It would be beneficial for the Foundation
to establish some estimate of expected effort
required by the awardee to support these
activities. For programs already in place wherein
project costing did not assume such required
support, consideration should be provided for
additional funding to address this additional
scope.
In addition, the BSR explanation notes that the
BSR "...determines if these policies and
procedures conform to OMB requirements, NSF
expectations, and other applicable federal
regulations." While the OMB requirements and
federal regulations are explicit documents, it is
not clear where "NSF expectations" are defined.
This lack of clarity extends to the BSR Guide, and
needs to be addressed more specifically.
Paragraph on required use of the Financial Data
Collection Tool may need revision based on
readiness for use.
Please clarify whether the cost submission and
cost incurred audits apply to facilities in
construction, operations, or both. We suggest
both.
Will be assessed and addressed in
later version.
No change. Data Collection Tool will
be ready for use.
Addressed in this revision. Following
was added to Section 4.5.3.4: “These
incurred cost audits are for both
construction and operations awards.”
Page 52 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-152
C-153
C-154
LFM Section/
Paragraph
4.5.3
Recipient
Performance
Reviews and
Audits,
4.5.3.5
4.5.3
Recipient
Performance
Reviews and
Audits
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(OIG)
EVM section stated "This section
reserved for future content.
As we have recommended in the past, we
suggest the certification of EVM systems and the
validation of EVM data.
Will be assessed and addressed in
later version after the finalization of
the SOG.
NSF
(DFM)
4.5.3 Recipient Performance
Reviews and Audits:
No change. Payment testing is done
as part of the BSR. LFM already makes
note of the BSR. BSR Guide is on LFO
Public Website. Add this reference to
the footnote.
4.5.3.1
NSF
(MPS)
Recommend considering adding Reviews for
Payment Testing and for Program income
Reporting. This will increase awareness of these
monitoring processes and establish expectations
for recipient responses to the reviews. We can
provide language should this be recommendation
be accepted.
4.5.3.1 misuses a lot of terms and creates
confusion. It refers in a single sentence to
conducting reviews and internal audits in
conformance with generally accepted
accounting, project management, and
operations management standards and
practices. Further in, it talks about safety
reviews, as well as many technically oriented
reviews. Financial reviews of awardees, and
the internal audit and processes of the
awardee, is a huge subject that should not be
addressed here. Safety reviews of the awardee
organization are similarly the subject of state
regulation, OSHA, and other standards. Often,
awardees are state universities subject to
regulation and oversight.
Content & Clarify
Addressed in this version. Language in
Section 4.5.3 revised to clarify
Recipient Internal Reviews and NSF
External Reviews.
Page 53 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
Source
LFM Language/ Observations
Comment
NSF Response
C-155
LFM Section/
Paragraph
4.5.3.2
External
Reviews
NSF
(MPS)
“The PO will typically use a
standard review ‘template.’
These well-defined review
formats provide a broad outline
against which the project can
be compared and checklists
that can be used to assess the
status of the project.” A
footnote regarding the
template says: “Please contact
the cognizant NSF PO for details
and a description of best
practices and/or preferred
templates.”
Text makes no distinction between operations
and construction reviews. To be effective, the
Program Officer should have in-depth
professional knowledge of the scientific
operation of the facility, and should develop a
review charge written to elicit advice matched
to the specific needs and challenges of the
facility. The review should not be based on a
check-the-box template. The PO should consult
with the DD, G/AO, DD, and LFO (where
appropriate) to develop the charge.
Addressed in this version. Language
revised in Section 4.5.3.2 to indicate
the review charge should be matched
to the needs of the facility at the
time.
C-156
4.5.3.2
NSF
(MPS)
Addressed in this version. Language in
Section 4.5.3.2 revised to “The PO
may use a standard review
“template” developed by the Division
or Directorate. “
C-157
4.5.3.2
NSF External
Reviews.
NSF
(OIG)
LFO seems to be included in all operations
reviews and has the right to concur on all
operational review charges. It claims the PO
will use a standard review template. It claims
that exceptional circumstances may require
the PO to consult with LFO Liaison to constitute
a review charge and format to meet the
specific requirements of the review.
We suggest review of updated vendor
quotes/bids be added to the circumstances. As
also referred to in Section 4.2.5.7, we suggest the
NSF clearly define scope, change in scope and
major change in scope, to avoid the possibility of
subsequent disagreement between NSF and its
recipients as to the meaning of these terms.
Content & Clarify
Additional ad hoc reviews may be
requested by the PO under
certain circumstances, such as
significant re-planning of
construction projects, changes in
key personnel, and major
changes in research technical
design, direction and scope.
Addressed in this version. Language in
Section 4.5.3.2 revised. The scope is
articulated in the WBS and scope
management plan.
Page 54 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
4.5.5
Re-baselining
Source
C-159
4.5.5
Re-Baselining
NSF
(OIG)
C-160
4.6
NSF
(GEO/
OCE)
C-161
5.2
NSF
(GEO/
OCE)
C-158
Content & Clarify
LFM Language/ Observations
NSF
(MPS)
The re-baselining section does
not explain what type of reviews
NSF will complete in the event of
a re-baseline. For example,
whether re- baselined proposals
will be required to go through an
independent cost review.
Comment
NSF Response
Every re-allocation of budget or schedule
contingency changes the baseline, as does the
use of scope contingency! Usually re-baselining
occurs when the underlying planning
assumptions are so different from project
reality that the major re-planning is needed.
What criteria trigger the re-baselining
definition? What should be done as a result?
This section has little information content, and
needs to be rewritten to provide substantive
definition.
We suggest clarifying what types of reviews rebaselined proposals will go through.
Addressed in this version. Language
was revised in Sections 2.4.1 and 4.5.5
to clarify re-planning and rebaselining.
Is there a definition for MOA? Sometimes MOUs
are between the Awardee and foreign countries,
this section indicates only NSF can have MOUs… I
think it depends on the ownership model and
other factors
Because expert judgement is required for risk
assessment, I recommend that the Awardee
assure the experts are vetted by their project
advisory committee and maybe even NSF prior to
conducting the assessment. This should be a
recommendation and not a requirement
Addressed in this version. Added
“with NSF” to clarify. PAM indicates
that MOU’s & MOA’s are synonyms.
PAM refers to only NSF MOU’s.
Addressed in this version. Language
added to Section 4.5.5 to indicate that
re-baselined projects may go through
external panel review, NSF cost
analysis, and Board Approval.”
Will be assessed and addressed in
later version.
Page 55 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
5.2.1
Introduction
Source
LFM Language/ Observations
Comment
NSF Response
NSF
(GEO/
PLR)
Comment indicated to add “and generally are not
amenable to automation” at the end of the
sentence.
No change. Comment is an efficiency
observation.
C-163
5.2.11.1
Contingency
Budget
Timeline
NSF
(OIG)
5.2.3.2
Contingency
Definitions
NSF
(OIG)
Consistent with our comments in 4.2.4, we
suggest that NSF instruct its recipients that
estimates based on vendor quotes/bids should be
kept current (i.e., preferably less than a year old
but no more than 18 months) throughout the
project life cycle at least until award.
As also mentioned in 4.2.5 .1, we suggest NSF
also state its expectation not only in terms of a
confidence level, but also the precision of the
estimate (so that NSF can say, for example, that it
is 80% confident that the point estimate is
accurate within a range of plus or minus 20%).
No change. Addressed in Section in
4.2.4.
C-164
C-165
5.2.3.2
Contingency
Definitions
5.2.6.3
Risk
Description
NSF
(GEO/
PLR)
NSF
(OIG)
3rd paragraph “ While
quantitative risk analysis has
become easier and more
sophisticated, it is unlikely to fully
replace qualitative risk analysis
because the quantitative analysis
requires validated inputs that are
more labor intensive to
produce.”
At the Final Design Review (FDR)
the PEP budget estimate should
be substantially based on
externally obtained cost
estimates (vendor quotes, bids,
historical data, etc.)
For MREFC construction projects,
the amount of budget
contingency is determined by
performing a probabilistic risk
analysis on the baseline cost and
schedule and selecting a Total
Project Cost with an acceptable
confidence level (typically
between 70-90%).
“Budget contingency is held
separately from the PMB and
allocations…
However, note that NSF does not
allow the use of contingency for
risks that are commonly referred
to as "unknown unknowns" such
as exceptional events or major
changes in scope.
PMB not defined in chapter
Addressed in this version. PMB is
defined as Performance
Measurement Baseline.
Addressed in this version. Language
has been added and revised
throughout the LFM on the scope
management plan and the usage of
contingency.
C-162
C-166
Content & Clarify
Suggest that NSF provide some examples of
exceptional events for which contingency is not
allowed to be used. Also, as mentioned in 4.2.5.7
and in 4.5.3.2, we suggest the NSF clearly define
scope, change in scope and major change in
scope, to avoid the possibility of subsequent
No change. Level of confidence in the
estimate changes as the design
develops. Uncertainty in the cost
estimate is included as part of the risk
register and part of the quantitative
risk analysis for development of the
budget contingency.
Page 56 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
LFM Section/
Paragraph
Source
LFM Language/ Observations
C-167
5.2.7.3
NSF
Limitations of (GEO/
Qualitative
PLR)
Analysis
2nd bullet: “They must also
evaluate whether other risks
might keep that risk from
creating much improvement if it
were mitigated.”
C-168
5.2.7.4
NSF
(GEO/
PLR)
C-169
5.2.8.8
Handling
Inflation
NSF
(OIG)
2nd paragraph, Page5.2.7-6:
“Research has shown that the
overlap in probability values with
common word definitions is
severe.”
Inflation is part of the NSF
budgeting and project planning.
C-170
5.5
Environment
al
NSF
(GEO/
OCE)
C-171
5.5
Environment
al
Consideratio
ns
NSF
(GEO/
PLR)
Content & Clarify
Comment
disagreement between NSF and its recipients as
to the meaning of these terms.
rephrase for clarity: ... keep mitigation of a
particular risk from resulting in much
improvement
Addressed in this version. Revised to:
“They must also evaluate whether
other risks might prevent mitigation
of a particular risk from resulting in
much improvement.”
Research has shown that the lack of overlap in
assigning probability values with common word
definitions is severe.
Addressed in this version.
Sentence revised according to
comment.
We suggest that NSF require its recipients to
affirm (and also verify in NSF reviews of proposal
budgets) that inflation is not double-counted in
other estimates, such as in contingency.
Addressed in this version. The method
for escalation is to be addressed in
the CEP and BOE. The project may
include a risk that the inflation is
different from the BOE.
Will be assessed and addressed in
later version. Environmental
compliance needs to be prior to
funding for construction. After
internal guidelines are complete, LFM
will be edited as accordingly.
Language added regarding potential
time associated with EA’s.
Addressed in this version. Per
guidance from OGC, ACA wasn’t
added but the following language
was: “While NEPA and its
implementing regulations focus on
activities that take place within the
United States, proposed activities that
Where are the decision milestones and off ramps
defined? Does a PEA need to be completed by
PDR and noticed? This is a lesson learned from
past projects.
2nd paragraph: “These statutes
include, but are not limited to,
the National Environmental
Policy Act (NEPA), the National
Historic Preservation Act (NHPA)
and the Endangered Species Act.”
NSF Response
Add
, the Antarctic Conservation Act,
Page 57 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt #
C-172
LFM Section/
Paragraph
2.4.1; 2.5.3;
5.4
External
reviews
Content & Clarify
Source
NSF
(LFO)
LFM Language/ Observations
Comment
It is not clearly stated that the reports from
external reviews are to NSF and not the recipient.
Recipients should not react to reports unless
directed to by written instruction by the PO
NSF Response
take place outside of United States
may also be subject to NEPA.”
Addressed in this version.
2.4.1-1 “Following internal NSF review
and evaluation of the external panel
report, NSF may issue written
guidance to the recipients for
subsequent response and action.“
2.5.3 “As a result of internal NSF
evaluation of the report and
assessments, the appropriate NSF
entity may issue written guidance to
the recipient for subsequent response
and action.”
5.4 “The reports and
recommendations from these
external reviews are made directly to
NSF. NSF evaluates the review panel
input, determines the appropriate
response, and issues written guidance
to award recipients for any
subsequent response and action.”
Page 58 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
TYPOS & EDITING COMMENTS TABLE
Page 59 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt
#
T-1
LFM Section/
Paragraph
2.1.3
Facility Life
Cycle
Source
T-2
2.1.6
Roles and
Responsibilities
NSF
(GEO/
PLR)
T-3
2.1.6 and
Summary of
Significant
Changes
NSF
(MPS)
T-4
2.1.6.4
Governing
Bodies
NSF
(GEO/
PLR)
T-5
2.4.1
Construction
Award
Management
and oversight
NSF
(GEO/
PLR
Typos & Editing
LFM Language/Observations
NSF
(OISE)
Page 2.1.6-11, bottom of page “This
responsibility include coordination of
planning”
Changed issuance of the HLFO
MREFC facility status report from
“monthly” to “periodic” on page
2.1.6-13.
Page 2.1.6-19 “The MREFC Panel …
makes recommendations to the
Director. The Panel consists of the
NSF Deputy Director (Chair), the Ads,
Program Office Heads”
2nd paragraph: “Following the review,
the PO and the LFO Liaison will each
independently assess the review,
confer on areas of concern, share
their views, and report their
observations through their
respective supervisory. “
Comment
NSF Response/Resolution
Pg. 22 - end of para 1 - "(ideally) were
partnerships" should be "(ideally) where
partnerships"
Addressed in this version. Typo
corrected.
Typo (“includes”)
Addressed in this version. Typo
corrected.
I did not find this. Whether or not HLFO
issues status reports, the spending rate
for MREFC projects motivates NSF
Program Officers and other staff
receiving technical and financial status
reports on a monthly basis during
construction.
Typo: capitalize d in ADs
Addressed in this version. Changed
to 2.1.6-15
Typo: insert “chain” at end of sentence
Addressed in this version. Typo
corrected.
Addressed in this version. Typo
corrected.
Page 60 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt
#
T-6
LFM Section/
Paragraph
2.4.2
Commissionin
g Plan
Source
LFM Language/Observations
Comment
NSF Response/Resolution
NSF
(GEO/
PLR)
Typo “who useful life” -> “the useful life of
which”
Addressed in this version.
Language revised.
T-7
3.1
Introduction
to
Management
Plans
NSF
(GEO/
PLR)
8th bullet:
“A discussion of how major
maintenance issues (such as
budgeting for periodic replacement
of long-lived capital assets who
useful life extends beyond the
duration of the CA) will be handled”
3rd paragraph: “An IMP also provides
financial strategies for funding given
the estimated budgetary estimates.”
Duplicated ‘estimates/d”
Addressed in this version.
Language revised.
T-8
3.4.1
NSF
(GEO/
PLR)
Should “complies” be “compiles”?
Addressed in this version. Typo
corrected.
T-9
3.5
Replace IT security with cybersecurity
Addressed in this version.
Language revised.
T-10
4.2.1 Cost
Estimating
and Analysis
Overview
NSF
(GEO/
OCE)
NSF
(GEO/
PLR)
Alternate text for the figure contains typo
“Brand Chief”
Addressed in this version. Typo
corrected.
Typos & Editing
Table 3.4.1-1
4.4 Scope Management Plan and
Scope Contingency
“….
Scope Contingency complies savings
from potential de-scoping options,
with decision points …”
Figure 4.2.1-1: Alternate text
Page 61 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt
#
T-11
LFM Section/
Paragraph
4.2.3.3
Source
LFM Language/Observations
Comment
NSF Response/Resolution
NSF
(GEO/
PLR)
editorial: add space before parenthesis and
"e.g. after; use only one set of parentheses
so separate two examples by comma
Addressed in this version. Typo
corrected.
T-12
4.5.3.1
Recipient
Internal
Reviews
NSF
(GEO/
PLR)
“ Guidance and examples of common
WBS elements can be adapted from
GAO, DOE, DOD and other guidance
and tailored for NSF projects(U. S.
Department of Energy WBS Handbook
dated August 16, 2012) (DoD Standard
Practice WBS for Defense Materiel
Items, MIL-STD-881C, dated 3 October
2011).”
“Although internal review team are
typically populated by project staff…”
Typo - teams
Addressed in this version. Typo
corrected.
T-13
4.6.1
Partnerships
Overview
NSF
(GEO/
PLR);
NSF
(OISE)
Last paragraph: “ The NSF Office of
International and Integrative Activities
(OIIA) should be advised”
(OISE) !
Pg. 157 – Please correct the name of the
office; “NSF Office of International and
Integrative Activities (OIIA)” should be “NSF
Office of International Science and
Engineering.”
Addressed in this version.
Language revised.
T-14
5.1
Introduction
NSF
(GEO/
PLR)
“They are based primary on current
standards and best practices for
project management.”
“primarily”
Addressed in this version. Typo
corrected.
T-15
5.2.6.1 Risk
Identification
Process
NSF
(GEO/
PLR)
Last paragraph: “The Risk Register
provides a means of tracking and
reporting status as risks occur and
migration strategies are implemented,
and is an important tool for Risk
Management implementation.”
Mitigation?, not migration
Addressed in this version. Typo
corrected.
Typos & Editing
Page 62 of 74
Public Comments on Large Facilities Manual (LFM) May 2016 & NSF Responses
Cmt
#
T-16
LFM Section/
Paragraph
5.6
Source
NSF
(MPS)
Heading 5.6 has a typo.
Addressed in this version. Typo
corrected.
T-17
Summary of
Significant
Changes
NSF
(MPS)
Change 5 refers to a third bullet on page
2.2.2-9, yet there is no page 2.2.2-9. (The
manual goes from 2.2.2-1 to 2.3.1-1)
Addressed in this version.
Corrected 2.2.2-9 to 4.2.2-9.
Typos & Editing
LFM Language/Observations
Comment
NSF Response/Resolution
Page 63 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
FINANCIAL DATA COLLECTION TOOL
COMMENTS TABLE
Page 64 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-1
F-2
Section/
Paragraph
Financial
Data
Collection
Tool:
General
Source
Financial
Data
Collection
Tool:
General
NSF (OIGAudit,
Maguire)
AURA
Financial Data Collection Tool
Comment
Are totals in this report intended to tie to any other report
submitted to NSF?
- Program reports include data on commitments which
are not reported here.
- ACM$ reports only show cash drawn, some expenses
reported here may not have been paid in cash yet.
- Audited financial statements
- How is program income reported?
We understand that the overall purpose of the reporting
tool is to provide for the collection of cost data from each
awardee for each large facility, to increase NSF's
management and financial oversight of its large research
facilities. Collection of this information will provide NSF
valuable budget and expenditure information needed for
NSF's oversight purposes. However, we note overall that the
reporting tool is not the equivalent of an incurred cost
submission (which, among other things, reports total
recipient direct costs for all of a recipient's awards and
calculates and proposes indirect cost rates subject to
negotiation). Therefore, in terms of facilitating an incurred
cost audit, it seems that the reporting tool, at best, could
only facilitate a direct cost audit with a review of the
application of indirect cost rates of the award being
reported. NSF would have to further coordinate with the
awardee for an indirect cost rate audit for an NSF cognizant
awardee.
Nevertheless, we are offering comments to help ensure that
NSF' s policies and procedures will provide reasonable
assurance that the controls in place, both during
construction and operations, will strengthen NSF's and its
awardees' ability to exercise proper stewardship of large
NSF Response/Resolution
Addressed in this version. This data tool does not
tie to any other report submitted to NSF.
Addressed in this version. Updated Instructions
(Overview) that Program Income is not included in
the Data Tool. A description has been added to the
instructions which state that Program Income
should not be included in your costs but provided
on the Program Income Reporting Worksheet per
the PAPPG.
No change. Comment is a preamble to other
comments.
No change. NSF will only be collecting this
information for the purposes of conducting
incurred cost audits. Additional NSF purposes have
not been identified at this time.
Recipients are allowed to re-budget cost categories
without previous approval from NSF. Only, budget
reallocations associated with changes in scope
need approval.
NSF does not intend to conduct incurred cost
audits of indirect costs.
Page 65 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
Section/
Paragraph
Source
Comment
NSF Response/Resolution
facility funds. Our comments are provided below by
section. We hope you find them useful.
F-3
Financial
Data
Collection
Tool:
Instructions
AURA
The instructions on the form are unclear as to whether a form
for each CSA under a CA is to be submitted or if one form for fill
CSAs under a CA is required.
Additionally, the guidance does not provide specific deadlines
requirements. Currently, it references to the award terms and
conditions. However, the award terms and conditions do not
address this compliance requirement.
It is not yet clear when the requirement for submitting the
'first' report is, if for FY16 data that could create issues. NSF is
asking for public comment that will certainly lead to some
changes in the data collection tool and will take time for NSF to
revise. Given that we are already into the FY 4th quarter, there
likely will not be sufficient lead time to properly prepare once
the revised tool is released.
Financial Data Collection Tool
Addressed in this version. The instructions were
strengthened to make it clear a new form must be
created for each CSA. There should be only one CA
for each submission of this data tool.
Addressed in this version. The instructions have
been updated to give guidance on the deadline for
annual submissions. The Terms and Conditions will
also stipulate the frequency which will be
determined by the Grants and Agreements Officer,
and the Cooperative Agreement’s Terms and
Conditions. It will be no less frequently than
annually 60 days after the end of the current award
funding year (performance period) to NSF.
Addressed in this version. Clarifications were made
in the LFM on the annual due date of the tool and
in the Terms and Conditions.
The Instructions were updated to require this data
collection annually 60 days after the end of the
current award funding year (performance period)
to NSF as well in the Terms and Conditions of the
award.
Page 66 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-4
Section/
Paragraph
Financial
Data
Collection
Tool:
Instructions
and
Instructions
Overview
Source
NSF (OIGAudit,
Maguire)
Comment
Overall, the reporting tool collects budgeted and actual
costs for individual Cooperative Agreements
(CA)/Cooperative Support Agreements (CSA). However, NSF
does not clearly explain that the reporting tool also provides
for budget to actual comparison/reconciliation at the total
project level, which is the level cumulative proposed
budgets are made and National Science Board approval is
obtained. Therefore, the instructions should be clarified to
explain that NSF intends to fully account for all of the
awards made for each NSF large facility that are in total
valued at $100 million or greater and may be comprised of
several different CSA and CA awards, funded by multiple
NSF appropriations.
Additionally, the reporting tool does not collect
"Contingencies" or "Fee" data (either in budget or in
expenditures to date). The tool also does not collect
drawdown data or cash on hand data that could be
reconciling items when comparing budget and actual costs
and costs claimed to the general ledger. Without this
information it will not be possible to adequately reconcile
and compare actual to budgeted costs to determine how
funds are being spent in comparison to how they were
planned.
NSF Response/Resolution
Addressed in this version. The Instructions have
been revised to better clarify NSF’s intention to
fully account for all of the awards made for each
NSF large facility that are in total valued at $100
million or greater and may be comprised of several
different CSA and CA awards, funded by multiple
NSF appropriations.
Recipients are allowed to re-budget cost categories
without previous approval from NSF. Only, budget
reallocations associated with changes in scope
need approval.
Budget to actual comparisons will not provide a
reconciliation for the project.
Addressed in this version.
The Instructions have been revised to indicate that
Contingencies" or "Fee" should be entered in the
Other Direct Costs and to add a note.
No change.
This tool will not capture drawdown or cash on
hand data. This data is already captured in a
Federal system (ACM$) and it is not required to
submit this data twice. This data can be obtained
through the ACM$ system.
The tool was developed to capture incurred cost
data, not to collect cash drawdown or cash on
hand information. Cash drawdown is available in
ACM$.
Financial Data Collection Tool
Page 67 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-5
Section/
Paragraph
Financial
Data
Collection
Tool:
Instructions
Overview
Source
NSF (OIGAudit,
Maguire)
Financial Data Collection Tool
Comment
In the Overview Section, specific criteria for applicability of
the reporting tool should be clarified, i.e. the tool is
required for each CSA and CA for a facility or project that
has a total awarded amount of $100 million or more. It
should explain that NSF will identify all the awards that
comprise each project and modify the award agreements to
identify each CSA and CA that require completion of the
reporting tool. This section should state that one other
purpose of the tool is to provide for a comparison of
expenditures to budget amounts for each award and the
total of the project's awards, to the maximum extent
practicable.
Also, the Instruction Overview states, "Only this NSFapproved tool should be used to submit incurred cost
data." This statement should be clarified because it could
create confusion if a full incurred cost submission is
required from an awardee for which NSF is cognizant for
the purposes of negotiating the awardee's indirect cost
rate.
NSF Response/Resolution
Addressed in this version. A clarification was made
to the instructions on the required CA and CSA’s.
To be addressed in this version.
NSF will revise the language and give clarity.
No change. This tool will not be used as a
comparison tool but to provide the Auditor with
basic information on number of awards, amounts,
expenditures, subcontracts, subawards and
indirect cost rate information.
Addressed in this version. Changes made to
Instructions on this sentence and it has been
removed.
NSF does not anticipate obtaining other incurred
cost submissions.
Page 68 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-6
Section/
Paragraph
Financial
Data
Collection
Tool:
Federal
Register
Notice
Source
NSF (OIGAudit,
Maguire)
Financial Data Collection Tool
Comment
The Comment Request Summary states that NSF will be
"requesting OMB clearance of this collection for no longer
than 3 years." We suggest that NSF also state its intentions
after that time period (in a later version of the reporting
tool), especially since the subsequent charge statement is
more focused on determining and assessing awardee
burden, rather than on emphasizing NSF's need to improve
its, and the awardee's, financial management of large facility
awards. The instructions should explain that the reporting
tool was instituted to ensure that funds for large facility
awards. The instructions should explain that the reporting
tool was instituted to ensure that funds for large facilities
are properly expended and managed. This message was
verbally conveyed during NSF's recent Large Facility
Workshop, but was not communicated as well in the written
request for comments from the community.
NSF Response/Resolution
No change. Since this tool may change as it is used
NSF will not state any intentions of other uses of
the tool at this time.
Addressed in this version. NSF has revised the
language and gives clarity on the use of the tool
and added this to the Instructions.
Page 69 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-7
Section/
Paragraph
Financial
Data
Collection
Tool: Setup
Tab
Source
AURA
Financial Data Collection Tool
Comment
The cell in which the Users is to fill in the number of
Subcontracts or Subawards being reported will not take a
number greater than 7.
There will likely be situations where there are more than 7
contracts totaling a minimum of $150,000. Either remove the
limit and allow input for all contracts above 150k or explicitly
state if contracts should be chosen by highest BAC, highest
BCWS, highest Actuals, etc. for the 7 being chosen.
For grand totals to tie out for the total project all subcontract
values should be used. Only selecting contracts totaling a
minimum of $150,000 could leave a substantial gap in reporting
the total BAC of the project and may cause confusion validating
data. Perhaps a solution is one line for all other contracts less
than $150,000.
The input sheets need to be clearer if budgets being entered
are cumulative to date or BAC. Reducing the ambiguity of the
budget data being requested would be helpful (specifically call
out if this is a FY request, cumulative to date, or BAC). It may
be useful to populate the tool with a sample (test) data for
examples of how the budget and actual totals trace across the
worksheets.
There are some cases where cells should be merged on the
Subcontract/Sub award sheets (i.e. D:E, F:G, etc. for general
look and feel). It also seems like rows 31 and 32 on those
sheets should be formulas from the data entered above (Gray).
Just a note, there could be a disconnect with this tool and
EVMS data as EVMS data also includes estimated actuals based
on delayed accounting processing of invoices.
NSF Response/Resolution
Addressed in this version. Clarification was made
to explain that all subcontracts and subawards
would be required in the Data Tool.
Addressed in this version. Clarification was made
to explain that only subawards and subcontracts
over $150,000 would be entered as separate field
and one extra would be entered for all remaining
subawards and subcontracts under $150,000 as a
combined and cumulative entry.
Addressed in this version. NSF has clarified that
budgets and expenditures are cumulative to date
for the current year.
Addressed in this version. Cells have been merged.
No change. This data tool is not requiring EVMS
data.
Page 70 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-8
Section/
Paragraph
Financial
Data
Collection
Tool:
CSA Budget,
Cost, and
Program
Expenditures
Worksheets
Source
NSF (OIGAudit,
Maguire)
Financial Data Collection Tool
Comment
Under major costs elements, the tool does not provide lines
for "Contingencies" or "Fee" (either in budget or in
expenditures to date), although these may be significant
items included in some CAs. Contingencies are required
budget information for construction awards specifically
listed on Form SF 424C (line 13), which NSF has agreed to
use and should be collected by this tool. Fee should also be
included in the tool if a recipient proposes fee, since this is
also a cost to NSF. Additionally, we note that the tool does
not provide for cost drawdowns (ACM$), which should be
collected for comparison to costs claimed.
The instructions for worksheet preparation for the CSA Cost
Worksheet state that the Program Charges Worksheet
consolidates this data for all CSAs. There is no Program
Charges Worksheet, so the instructions should be clarified
(perhaps the Program Expenditures Worksheet).
It appears, based on the color coding legend provided, that
the current expenditure information in the CSA Budget
Worksheet is linked from other report tabs and the current
claimed amount on the CSA Cost Worksheet is autocalculated. The tool should explain this and caution the
awardee to ensure that the calculated expenditure amount
agrees with the amounts claimed. Also, the reporting tool
should provide for an explanation in the event there is a
difference between the General Ledger and Job Cost Ledger
on the CSA Cost Worksheet.
It would also be useful if the tool used consistent terms
throughout the different worksheets to identify current
costs claimed/expenditures and to ensure that the title of
the worksheet tab matches the title on the worksheet.
NSF Response/Resolution
Addressed in this version. The Instructions have
been revised to indicate that Contingencies" or
"Fee" should be entered in the Other Direct Costs
and to add a note.
No change. NSF does not use the SF-424C budget
document but the NSF-1030. NSF has provide
guidance on were to include both "Contingencies"
or "Fee" on the approved budget.
NSF has replaced the word charges from the
Instructions.
Addressed in this version. Added to cost and CSA
budget worksheet and the instructions
Addressed in this version. NSF has added
clarification on how auto-calculated sections work
and edited the instructions and added the color
coding info.
Addressed in this version. NSF has provided more
written clarity in the event there is a difference
between the General Ledger and Job Cost Ledger
on the CSA Cost Worksheet. Added to instruction:
If there is a difference between the General Ledger
and Job Cost Ledger on the CSA Cost Worksheet
provide an explanation in the comments field.
Addressed in this version. NSF edited tabs and
fields to be consistent.
Page 71 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-9
F-10
F-11
Section/
Paragraph
Financial
Data
Collection
Tool:
Subcontract
&
Consultants
Worksheets
Financial
Data
Collection
Tool:
Subawards
Worksheet
Financial
Data
Collection
Tool:
Indirect Cost
Rate
Worksheet
Source
NSF (OIGAudit,
Maguire)
Comment
NSF Response/Resolution
We note that worksheet does not require a specific
classification of subcontracts vs. consultants. This distinction
is important and needs to be made for indirect rate
application purposes because modified total direct costs
(MTDC) include the first $25,000 of sub
grants/Subcontracts, while the remaining portion of
Subgrants/ Subcontracts over $25,000 is excluded.
Additionally, the worksheet does not request the type of
subcontract (CPFF, firm fixed-price, etc.). It appears the
worksheet only provides for cost reimbursement
subcontracts, but there is no place for profit which should
be collected because it is part of total cost. There should
also be a place on the worksheet for firm fixed-price
subcontracts. Finally, the worksheet should also include a
checkbox to document whether the Uniform Guidance
required cost/price analyses in excess of the Simplified
Acquisition Threshold (currently $150,000) are maintained
in the file and are available upon request.
No change. This data tool is to provide basic
information to the Auditor. No listing of MTDC will
be added at this time.
NSF (OIGAudit,
Maguire)
This worksheet should also include a checkbox to document
whether the Uniform Guidance required cost/price analyses
in excess of the Simplified Acquisition Threshold (currently
$150,000) are maintained in the file and are available upon
request.
No change. NSF will not require any checkbox for
recipients to state if they are over the Simplified
Acquisition Threshold. All documentation on
contracts and subawards are required for an
Incurred costs audit but not for this tool.
NSF (OIGAudit,
Maguire)
We note that worksheet provides "rate applied" and "total
expended," but does not request the type of rate applied
(e.g., provisional, predetermined fixed, carry forward, etc.),
which should be requested. Also, the worksheet only
addresses three types of indirect rates (classified as F&A,
G&A and ID other). Some organizations may have additional
indirect rates that should be disclosed, so the tool should be
modified accordingly.
Will be assessed and addressed in later version.
NSF has a field for type of rate and a place for
other types of cost rates to be listed.
Financial Data Collection Tool
Addressed in this version. NSF clarified the use of
consultants and where this info is captured. New
contract categories were added to included Fixed,
Performance based, Cost Reimbursable and T&M.
Page 72 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-12
Section/
Paragraph
Financial
Data
Collection
Tool:
Certification
of Costs
Source
Comment
NSF (OIGAudit,
Maguire)
Suggest expanding certification to include award terms and
conditions, NSF policies and recipient policies.
No change. Certification is only for the data
supplied. Recipient has already certified to follow
Federal regulations and Terms and Conditions
when applying and accepting the award.
F-13
Financial
Data
Collection
Tool: CSA
Budget
Worksheet
AURA
Guidance needed for the level of supporting documentation
detail required to provide for "total expenditures to date"
on existing CSAs. Is the first report submitted supposed to
report inception to date expenses as the current year
expenses?
No change. All expenditures must have detailed
supporting documentation per the Uniform
Guidance. This documentation is not required at
the time of the submission of this tool but at the
time of the Incurred Cost Audit. The first report
will begin at the end of the current funding year.
On the CSA Budget worksheet, it is unclear if this should
only include the 7 Subawards listed from previous sheets or
if this is a grand total. This is the problem with only entering
a subset of data for the Subcontract/Subawards as there will
be a delta between total budget numbers for the CSA and
the numbers displayed in other sheets.
F-14
Financial
Data
Collection
Tool:
Instructions
Cell 10
AURA
Financial Data Collection Tool
Instructions do not clearly define if awardees are expected to
insert summarized expense total in the cells Users are to
complete, or if detail (raw data) behind the summarized
expense totals is to be provided. The draft spreadsheet allows
for new tabs to be added which could facilitate the importing
of detail, but it is unclear if the draft spreadsheet works in the
same manner the final version will.
NSF Response/Resolution
Addressed in this version. NSF has expanded the
tool to include more than 7 subawards or
subcontract.
Addressed in this version. NSF has clarified that it
will be summarized data (no details or raw data).
Page 73 of 74
Public Comments on Financial Data Collection Tool (May 2016) & NSF Responses
Cmt
#
F-15
Section/
Paragraph
Financial
Data
Collection
Tool:
Instructions
Cell 12
Source
F-16
Financial
Data
Collection
Tool:
Subcontract
&
Consultants
and SubAwards
Worksheets
AURA
AURA
Financial Data Collection Tool
Comment
Cell 12 - This cell implies that multiple CSAs should be reported
in the spreadsheet tool and the data will be consolidated on
this worksheet. Conversations with NSF personnel at the Large
Facility Workshop indicate that a separate spreadsheet tool
should be submitted for each separate CSA. Clarification is
required on this matter. Please note that AURA has separate
CSAs under the same NSO CA that are funded from NSF funds
and MREFC funds and we suspect these should not be reported
as a combined total.
-Data as outlined on contracts is not required from the vendors
in this format, so now requiring this data would result in
additional cost for both the awardee and the vendor.
-Having this kind of detail in the proposal\budget stage is not
part of current process.
-Having to gather data for each contract paid during the
reporting period could result in an unmanageable amount of
data.
-Not all contracts provide this information, requiring it could
cause additional expense raising prices of services or goods.
-Many contractors that we work with are for profit companies
so their costs may not reflect the price we pay for services; how
do we report profit?
-Verification that the data is accurate will be difficult or costly.
-Time involved in gathering, storing and organizing this data
would require additional effort by the awardee.
NSF Response/Resolution
Addressed in this version. NSF has removed the
word consolidated as a separate CSA sheet will be
added for each CSA.
Addressed in this version. NSF provided more
clarity on the required contract information
submitted on the instructions tab.
Addressed in this version. Recipients are required
to track the expenditures for all subrecipients but
NSF is only requesting use of the cost categories
for subcontracts, consultants and subawards if
they match. If recipients track subrecipient funds
using other cost categories then add to the ODC
cost field and add a note in the Recipient
Preparation Notes.
Page 74 of 74
| File Type | application/pdf |
| Author | McGovern, Jean M. |
| File Modified | 2016-12-19 |
| File Created | 2016-12-16 |