Download:
pdf |
pdfEXECUTIVE OFFICE OF THE PRESIDENT
OFFICE OF MANAGEMENT AND BUDGET
WASHINGTON, D.C.
20503
July 10, 2019
M-19-23
MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES
FROM:
SUBJECT:
, Rus_sell T_. Vought ~
Actmg Director
\J
·J _~
L-
'-'-"-V- '
Phase 1 Implementation of the Foundations for Evidence-Based Policymaking
Act of 2018: Leaming Agendas, Personnel, and Planning Guidance
Background
The Foundations for Evidence-Based Policymaking Act of 2018 ("Evidence Act"), 1 signed into
law on January 14, 2019, emphasizes collaboration and coordination to advance data and
evidence-building functions in the Federal Government by statutorily mandating Federal
evidence-building activities, open government data, and confidential information protection and
statistical efficiency. Evidence is broadly defined and includes foundational fact finding,
performance measurement, policy analysis, and program evaluation (see Appendix A). 2 In
addition, the Evidence Act mandates a systematic rethinking of government data management to
better facilitate access for evidence-building activities and public consumption.
Despite previous efforts and resource commitments, Federal agencies often lack the data and
evidence necessary to make critical decisions about program operations, policy, and regulations,
and to gain visibility into the impact of resource allocation on achieving program objectives.
Investing in and focusing on the management and use of data and evidence across the Federal
Government will enable agencies to shift away from low-value activities toward actions that will
support decision makers: linking spending to program outputs, delivering on mission, better
managing enterprise risks, and promoting civic engagement and transparency.
The Evidence Act builds on longstanding principles underlying Federal policies and data
infrastructure investments supporting information quality, access, protection, and evidence
Pub. L. No. 115-435, 132 Stat. 5529.
Pursuant to the Evidence Act, "evidence" is defined as "information produced as a result of statistical activities conducted for a
statistical purpose." See 44 U.S.C. § 3561(6). The term " statistical activities" means "the collection, compilation, processing, or
analysis of data for the purpose of describing or making estimates concerning the whole, or relevant groups or components
within, the economy, society, or the natural environment and . . . includes the development of methods or resources that support
those activities, such as measurement methods, models, statistical classifications, or sampling frames. " See id.§ 3561(10). The
term "statistical purpose" means "the description, estimation, or analysis of the characteristics of groups, without identifying the
individuals or organizations that comprise such groups and ... includes the development, implementation, or maintenance of
methods, technical or administrative procedures, or information resources that support [those] purposes. " See id. § 3561(12).
1
2
building and use. 3 While anchored on these principles, the Evidence Act creates a new paradigm
by calling on agencies to significantly rethink how they currently plan and organize evidencebuilding, data management, and data access functions to ensure an integrated and direct
connection to data and evidence needs. This paradigm requires engagement and cooperation
from multiple actors within agencies (e.g., senior leadership, policy officials, program
administrators, performance managers, strategic planners, budget staff, evaluators, analysts,
front-line staff, and data professionals) and key external stakeholders. It accelerates, expands,
and reinforces the implementation of many activities underway, including the Federal Data
Strategy ("Strategy"); 4 the government-wide reorganization proposals to strengthen Federal
evaluation and reorganize the Census Bureau, Bureau of Labor Statistics, and Bureau of
Economic Analysis to improve economic statistical data; 5 new monitoring and evaluation
guidelines for agencies that administer foreign assistance; 6 new centralized mission support
shared services and standards; 7 and ongoing open data policy related to open data laws like the
DATA Act and Geospatial Data Act. 8
See, e.g., Treasury and General Government Appropriations Act, 2001 , Pub. L. No. 106-554, § 515(a) (2000); Confidential
Information Protection and Statistical Efficiency Act of2002 (" CIPSEA"), Pub. L. No. 107-347, § 502(9)(A)); Data.gov,
available at https ://www.data.gov (first national data catalog, established in 2009); Exec. Order No. 13,642, Making Open and
Machine Readable the New Default for Government Information, 78 FR 28111 (May 9, 2013), available at
https ://www.federalregister.gov/documents/2013/05/ 14/2013-11533/making-open-and-machine-readable-the-new-default-for•
government-information; Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity oflnformation
Disseminated by Federal Agencies, 67 FR 8452 (Feb. 22, 2002), available at
https ://www.federalregister.gov/documents/2002/02/22/R2-59/guidelines-for-ensuring-and-maximizing-the-guality-objectivityutility-and-integrity-of-informat; Office of Mgm't & Budget, Exec. Office of the President, 0MB M-13-13 , Open Data PolicyManaging Information as an Asset (2013), available at
https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/20 I 3/m-13-13 .pdf; Office of Mgm't & Budget, Exec.
Office of the President, 0MB M-13-17, Next Steps in the Evidence and Innovation Agenda (2013), available at
https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/20 I 3/m-13-17.pdf; Office of Mgm 't & Budget, Exec.
Office of the President, 0MB M-14-06, Guidance for Providing and Using Administrative Data for Statistical Purposes (2014),
available at https ://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2014/m-14-06.pdf; and Office of Mgm't &
Budget, Exec. Office of the President, Statistical Policy Directive No. I : Fundamental Responsibilities of Federal Statistical
Agencies and Recognized Statistical Units 79 FR 71610 (Dec. 2, 2014), available at https://www.govinfo.gov/content/pkg/FR2014-12-02/pdf/2014-28326.pdf.
4 See Office ofMgm ' t & Budget, Exec. Office of the President, 0MB M-19-18, Federal Data Strategy-A Framework for
Consistency (2019), available at https://www.whitehouse.gov/wp-content/uploads/2019/06/M-19-18.pdf.
5 See Office of Mgm 't & Budget, Exec. Office of the President, Delivering Government Solutions in the 21st Century Reform
Plan and Reorganization Recommendations (2018), available at https ://www.performance.gov/GovReform/Reform-and-ReorgPlan-Final.pdf.
6 See Office ofMgm ' t & Budget, Exec. Office of the President, 0MB M-18-04, Monitoring and Evaluation Guidelines for
Federal Departments and Agencies that Administer United States Foreign Assistance (2018), available at
https ://www.whitehouse.gov/wp-content/uploads/2017/l l/M-18-04-Final.pdf.
7 See Office ofMgm 't & Budget, Exec. Office of the President, 0MB M-19-16, Centralized Mission Support Capabilities for the
Federal Government (2019), available at https ://www.whitehouse.gov/wp-content/uploads/2019/04/M-19-16.pdf.
8 Digital Accountability and Transparency Act of 2014, Pub. L. No. 113-101 , 128 Stat. 1146, available at
https ://www.congress.gov/l 13/plaws/publl 01/PLA W-1 I 3publ 101.pdf; Geospatial Data Act of 2018, Pub. L. No. 115-254, div.
B, tit. VII, Subtit. F, 132 Stat. 3413, available at https://www.congress.gov/115/bills/hr302/BILLS-I 15hr302enr.pdf.
3
2
Iterative and Coordinated Implementation
Meeting the requirements of the Evidence Act will require ongoing, focused effort. 0MB
recognizes that implementation will necessarily build on activities already underway and, in
some cases, require coordination of these activities in new ways. 9 0MB is committed to: (1)
aligning related data and information policy guidance, such as the Strategy and related 0MB
guidance, like 0MB Circular A-11, as applicable; and (2) providing iterative and supplemental
guidance in four phases (see Figure 1 below): 10
•
•
•
•
Phase 1:
Phase 2:
Phase 3:
Phase 4:
Learning Agendas, Personnel, and Planning
Open Data Access and Management
Data Access for Statistical Purposes
Program Evaluation
See, e.g. , CIPSEA; Data.gov; Exec. Order No. 13,642; 0MB M-13-13; 0MB M-13-17; 0MB M-14-06; Statistical Policy
Directive No. 1; Comrn'n on Evid.-Based Policymaking, The Promise ofEvidence-Based Policymaking (2017), available at
https://www.cep.gov/cep-final-report.html (bipartisan report which provided the impetus for Pub. L. No. 115-435).
1 Cf Office ofMgm't & Budget, Exec. Office of the President, 0MB Circular No. A-130, Managing Information as a Strategic
Resource, available at https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/circulars/ A 130/al 30revised.pdf; Office of
Mgm' t & Budget, Exec. Office of the President, Management of Federal Information Resources (Dec. 12, 2000) (revising 0MB
Circular No. A-130), available at https://www.whitehouse.gov/wp-content/uploads/2017 /11 /Management-of-FederalInformation-Resources-Dec 12-2000.pdf; Office of Mgm't & Budget, Exec. Office of the President, 0MB Circular No. A-11 ,
Preparation, Submission, and Execution of the Budget (June 20 I 9), available at https ://www.whitehouse.gov/wpcontent/uploads/2018/06/al 1.pdf; 0MB M-13-13; and 0MB M-14-06.
9
°
3
Figure 1: Evidence Act Guidance
Phase 1:
Learning Agendas,
Personnel, & Planning
o
Learn i ng Age nd as
Ch ief Data O' i cers
Eval uati on O i cers
Stati sti ca l Officials
Agenc y Data Governa nce
Boar ds
Ch i e Data Offi ce r
Counc il
Eval uation O i ce r
Counc il
l nter agency Counc il on
St at i st ica l Po licy
Agency Eva luation Pl ans
Capac ity Assessments
Open Data Pl ans
Learning Agendas,
Personnel, & Plannin
January 14, 2019
Ev idence Act si gned
i nt o l aw
Phase 2:
Open Data Access
& Management
o
Phase 3:
Data Access for
Statistical Purposes
Make Data Open by
De aul t
Compr ehensive Data
Invent ory
Feder al Data
Cata logue
Repos itory of oo l s
and Best Practi ces
Phase 4: Proeram
Evaluation
Make Agency Data
Assets Av ail ab l e to
Stati sti ca l Un its
Expand Users' Secure
Access to Data Assets
through Stati stica l Un its
All ow Recogn i t ion o
New Stati sti ca l Un its
Standard i ze App licati on
Pr ocess or Access i ng
Dat a Assets
Cod ify St ati stica l Un it
Respons i bil" ies
Open Data Access
& Management
Program Eval uati on
Standards and Best
Practices
Pr ogr am Eva l uati on
Skills and
Competenc ies (with
OPM )
Data Access
for Statistical
Purposes
July 13, 2019
January 14, 2020
La w takes e ect
One Yea r Pos Ena ctment
12 M onths
6 M onths
4
Ongoing
Implementation &
Reporting
o
o
o
Budget Cycl es
Inform at ion Resou r ce
M anagement Pl ans
Perform an ce Pl ans
Strategic Plans
Fed er al Dat a Strat egy Annu al
Act ion Pl ans
Bienn i al Report
St at i st ical Progr ams Annua l
Report
Regu l atory Pl ann i ng
Inform at i on Collect i on Rev iew
Ongoing
Implementation
& Reporting
24+ M onths
Phase 1: Learning Agendas, Personnel, and Planning
This memorandum includes the key requirements for implementation of Phase 1 that agencies
should begin working on immediately (see Table 1 below). Phase 1 implementation includes
developing Leaming Agendas, designating and developing roles and responsibilities for
Personnel, and undertaking Planning activities.
I.
Learning Agendas: The Learning Agendas are, in many ways, the driving force for
several of the activities required by and resulting from the Evidence Act. The creation of
Learning Agendas requires agencies to identify and set priorities for evidence building,
in consultation with various stakeholders.
II.
Personnel: The newly-designated positions (Chief Data Officer, Evaluation Officer,
and Statistical Official) all play a key role in leading these activities, including
addressing the new reporting requirements. Their corresponding interagency councils
likewise support these functions and roles. These senior officials will also serve on a
newly required Data Governance Body inside their agency, which will set and enforce
priorities for managing data as a strategic asset to support the agency in meeting its
mission and, importantly, answering the priority questions laid out in the agency
Learning Agenda.
III.
Planning: The Evidence Act augments ongoing agency strategic planning activities to
promote better use and management of data and evidence, consistent with the GPRA
Modernization Act 11 and 0MB Circular A-11 Part VI and agency-specific management
routines, such as through data-driven performance and strategic reviews. Specifically,
Learning Agendas, which identify agencies' priority questions, drive related planning
activities, including:
o
o
o
Developing annual Evaluation Plans, which will summarize the specific
evaluations an agency plans to undertake to address those questions;
Undertaking Capacity Assessments, which, as part of agencies' strategic plans,
will help agencies to assess their ability and infrastructure to carry out evidencebuilding activities like foundational fact finding, performance measurement,
policy analysis, and program evaluation; and
Identifying the data needed to answer those questions.
Lastly, in tandem with Phase 2 guidance, Open Data Plans will inform dissemination
improvement priorities as part of agencies' Strategic Information Resources Management
Plans (discussed below).
The body of this memorandum summarizes these initial requirements, the corresponding
deadlines, reporting formats, and responsible officials. The appendices that follow provide
additional details on how agencies should implement these requirements, including how these
requirements tie into existing efforts, structures, and processes. Agencies should review and refer
GPRA Modernization Act of 2010, Pub. L. No. 111-352, available at https://www.congress.gov/111 /plaws/publ352/PLA WI I I publ352.pdf.
11
5
to all parts of this guidance, including the appendices, as they implement the Evidence Act's
requirements. 12 Agencies are expected to resource these activities appropriately.
Table 1: Evidence Act - Initial Implementation Requirements
Agencies:
Requirement
1
as
De\'elop Learnin .,,
o Ao.,encIa (E\'ldence-B
·
-. g ,\geull
u1·11c ·mg Pl an) ,e·,II.nm
.
Document Progress in Developing Learning Agenda
September 2019*
(in FY 2021 Evidence Template)
Submit Interim Learning Agenda
September 2020*
(concurrent with FY 2022 Annual Performance Plan)
Submit Annotated Outline of Learning Agenda
May 2021
(part of Initial Draft Strategic Plan)
Submit Full Draft Learning Agenda
September 2021 *
(part of Full Draft Strategic Plan)
Submit Final Draft Learning Agenda
December 2021
(part of Final Draft Strategic Plan)
Publish Final Learning Agenda
February 2022
(part of Final Strategic Plan)
Personnel
Name Members and Charter a Data Governance Body
D
Name Chief Data Officer
Name Evaluation Officer
Name Statistical Official
Participate in Designated Official Orientation
Participate in Interagency Councils
Responsible
Official(s)
Deadline
September 30, 2019
July 13, 2019
July 13, 2019
July 13 , 2019
September 2019
Ongoing
Planning
Develop Annual Evaluation Plan
Document Progress in Developing Evaluation Plan
(in FY 2021 Evidence Template)
Complete Evaluation Plan for FY 2022
(concurrent with FY 2022 Annual Performance Plan)
✓
**
EO
✓
**
EO
✓
**
EO
✓
**
EO
✓
**
EO
✓
**
EO
✓
✓
AH
✓
✓
✓
**
✓
✓
✓
✓
AH
AH
AH
CDO,EO, SO
CDO,EO, SO
September 2019*
✓
**
EO
September 2020*
✓
**
EO
✓
12 0MB recognizes that this guidance uses both the term "data" and the term "information," and that using both terms may cause
confusion. The Evidence Act introduced the term "data" into the statutory construct created by the Paperwork Reduction Act,
codified in subchapter I of chapter 35 of title 44 of the U.S. Code, which generally speaks in terms of "information." The
introduction of the term "data" in the Evidence Act did not, however, disturb the existing definitions or responsibilities that
involve the term "information." The result is that the law now describes agency responsibilities using both terms. In this
guidance, 0MB uses the terms "data" and "information" consistently with the source of the authority for each requirement.
13 See 31 U.S.C. § 90l(b).
6
Conduct Capacity Assessment
Propose Approach for Capacity Assessment
(in FY 2021 Evidence Template)
Submit Interim Capacity Assessment
(concurrent with FY 2022 Annual Performance Plan)
Submit Initial Capacity Assessment
(part of Initial Draft Strategic Plan)
Submit Draft Capacity Assessment
(part of Full Draft Strategic Plan)
Submit Final Draft Capacity Assessment
(part of Final Draft Strategic Plan)
Publish Final Capacity Assessment
(part of Final Strategic Plan)
September 2019*
✓
**
EO
September 2020*
✓
**
EO
May 2021
✓
**
EO
September 2021 *
✓
**
EO
December 2021
✓
**
EO
February 2022
✓
**
EO
See Forthcoming Phase 2 Guidance
* - Deliverable should be submitted at the time the agency submits information to 0MB per the Budget ,cycle
** - Strongly recommended for non-CFO Act agencies, as well as operational divisions, bureaus, and sub-agencies in CFO Act
agencies
EO = Evaluation Officer I AH = Agency Head I CDO = Chief Data Officer I SO = Statistical Official
Develop a Learning Agenda (Evidence-Building Plan) 14
I.
Each CFO Act agency should immediately start the process of developing and implementing a
multi-year learning agenda that coincides with the four-year timeframe defined for agency
strategic plans (see Appendix B for additional implementation details). As described further in
Appendix B, non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus
within CFO Act agencies are strongly encouraged to implement a learning agenda as appropriate
in coordination with the agency-wide learning agendas. Below are the key activities for agencies
to document progress in developing, publishing, and implementing a learning agenda:
•
•
•
FY 2021 Budget Submission (September 2019): Concurrent with their FY 2021 Budget
submissions in September 2019, agencies must document their progress in developing a
learning agenda in the Evidence Template that agencies are required to complete per
0MB Circular No.A-11.
FY 2022 Annual Performance Plan and Budget Submission (September 2020):
Concurrent with their FY 2022 Annual Performance Plan Submission in September 2020
(to coincide with the submission of the FY 2022 Budget), agencies must submit an
interim learning agenda. 15 Agencies are strongly encouraged to put their learning agenda
in place earlier and submit to 0MB prior to the September2020 deadline.
Draft Strategic Plan (May, September, and December 2021): As part of their initial draft
Strategic Plan submitted to 0MB in May 2021 , agencies must submit an annotated
outline of their updated learning agenda, building on the interim learning agenda
submitted in September 2020. The full draft learning agenda must be submitted in
14 A learning agenda is equivalent to the agency evidence-building plan required in Section JOI of the Evidence Act. 5 U.S.C.
§312(a).
15 Because this interim learning agenda is being developed during the last year of agency' s current strategic plan, the interim
learning agenda should be written to address short-term priority questions that can be answered more immediately and long-term
questions that are directly linked to the agency's core statutory mission and therefore are unlikely to differ substantively from one
strategic planning cycle to the next.
7
•
II.
September 2021 with the full draft Strategic Plan. The final draft learning agenda must be
submitted in December 2021 with the agency's final draft Strategic Plan.
Final Strategic Plan (February 2022): As part of their final Strategic Plan posted
publicly in February 2022, agencies must include their updated, final learning agenda,
which shall cover the same period.
Constitute Data Governance and Leadership
(a) Data Governance Body
By September 30, 2019, the head of each agency must establish an agency Data
Governance Body, to be chaired by the Chief Data Officer, with participation from relevant
senior-level staff in agency business units, data functions, and financial management (see
Appendix C). All agencies must include their Data Governance Body in their Strategic
Information Resources Management Plan's Governance sections as required by 0MB
guidance. 16 All agencies must report their Data Governance Body members to 0MB via
email at [email protected] no later than September 30, 2019.
(b) Designate Key Senior Officials
0MB recognizes that agencies have been working to put key senior officials in place to meet
Evidence Act requirements. As previously communicated over the past few months, the
deadlines for designating these positions are listed below.
Chief Data Officer - The Chief Data Officer (CDO) shall have authority and responsibility
for, among other things, data governance and lifecycle data management. The head of each
agency must designate a senior staff person in the agency with the required qualifications
(see Appendix C) no later than July 13, 2019. 17
Evaluation Officer - The Evaluation Officer shall have authority and responsibility for
providing leadership over the agency's evaluation and learning agenda activities. 18 The head
of each CFO-Act agency must designate, without regard to political affiliation, a senior staff
person in the agency with the required qualifications (see Appendix C) no later than July
13, 2019. Non-CFO Act agencies, as well as sub-agencies, operational divisions, and bureaus
of CFO Act Agencies are strongly encouraged to designate an Evaluation Officer as
appropriate.
Statistical Official - The Statistical Official shall have authority and responsibility to advise
on statistical policy, techniques, and procedures. The head of each CFO-Act agency must
designate a senior staff person in the agency with the required qualifications (see Appendix
C) no later than July 13, 2019. 19 Non-CFO Act agencies, as well as sub-agencies,
See 0MB Circular No. A-130, § 5.a.
44 u.s.c. § 3520.
18 5 U.S.C. § 313.
19 The Act requires the Statistical Official to be the head of an existing statistical agency in CFO Act agencies that have a
statistical agency. 5 U.S.C. § 314(a).
16
17
8
operational divisions, and bureaus of CFO Act Agencies are strongly encouraged to
designate a Statistical Official as appropriate.
All agencies must submit the names and contact information for their Chief Data Officer,
Evaluation Officer, and Statistical Official to [email protected] and post the names
on their https://[agency].gov/data webpage no later than August 2, 2019. Agencies must
provide any updates to these designations to 0MB via email at [email protected]
and update their webpages accordingly.
Evidence Act Orientation for Designated Officials - All designated officials must attend an
Evidence Act orientation hosted by 0MB in September 2019 to gain a fuller understanding
of the requirements under this guidance, develop a common baseline of knowledge about key
competencies required for their roles, learn about resources for fulfilling statutory
responsibilities, and meet and build relationships with their peers and with 0MB officials to
promote interagency coordination and collaboration.
(c) Interagency Councils
Designated key officials must also participate with their peers on the following interagency
councils:
Chief Data Officer Council
This Memorandum hereby establishes a Chief Data Officer Council (hereinafter "CDO
Council") that will include all agency Chief Data Officers, the Administrator of the Office of
Electronic Government (or designee ), the Administrator of the Office of Information and
Regulatory Affairs (or designee ), and an Ex Officio Member (to represent all Chief Information
Officers and Evaluation Officers). 0MB will designate a Chair and constitute a Charter for the
CDO Council. The Chair of the CDO Council, in consultation with the GSA Office of Executive
Councils, will develop an implementation plan for the Council that will consider the below
objectives, as well as lessons learned and best practices for managing Federal-wide executive
councils.
The CDO Council shall meet regularly to establish government-wide best practices for the use,
protection, dissemination, and generation of data; promote and encourage data sharing
agreements between agencies; identify ways in which agencies can improve upon the production
of evidence for use in policymaking; consult with the public and engage with private users of
Government data and other stakeholders on how to improve access to data assets of the Federal
Government; and identify and evaluate new technology solutions for improving the collection
and use of data. The CDO Council shares responsibilities with other government-wide councils
that conduct statutory data-related activities. 0MB expects that the activities of all these councils
will be coordinated through 0MB in order to assure that these activities are complementary and
carried out efficiently and effectively.
9
Interagency Council on Statistical Policy
The statutory purpose of the Interagency Council on Statistical Policy (ICSP) is to advise and
assist the Chief Statistician of the United States in the coordination of the Federal Statistical
System; the implementation of statistical policies, principles, standards, and guidelines; and the
evaluation of statistical program performance. 20 Its membership currently includes
representatives of 14 member agencies. These member agency representatives will be joined by
all statistical officials designated pursuant to section 2(b) of this guidance that are not already
members, increasing ICSP membership by 12. Consequently, the ICSP will include two
members each from the Department of Commerce and the USDA, which each contain two
principal statistical agencies, and one member each from all other CFO Act agencies.
The expanded membership will augment the ICSP's ability to meet its statutory purpose. In
addition, it will be a forum for collaboration, coordination, and information sharing among the
statistical agencies and additional statistical programs across all member agencies, including on
issues such as ensuring data quality and confidentiality, attaining and providing data access, and
playing an effective role in agency-wide data governance.
Evaluation Officer Council
The 0MB Evidence Team will establish and coordinate an interagency Council composed of
Evaluation Officers. The Evaluation Officer Council will serve as a forum to exchange
information, consult with and advise 0MB on issues that affect evaluation functions (e.g.,
evaluator competencies, program evaluation best practices, and evaluation capacity-building),
coordinate and collaborate on areas of common interest, and serve in a leadership role for the
broader Federal evaluation community. More information on the Council's composition and
membership is forthcoming.
ID.
Conduct Assessments and Planning
(a) Annual Evaluation Plan
CFO Act agencies shall immediately begin the process of developing their first Annual
Evaluation Plan. Non-CFO Act agencies, as well as sub-agencies, operational divisions, and
bureaus of CFO Act Agencies are strongly encouraged to develop an evaluation plan as
appropriate. At a minimum, the plan shall describe evaluation activities for the subsequent
year, including the key questions for each planned "significant" evaluation study, as well as
the key information collections or acquisitions the agency plans to begin. 21 Below are the
required activities by which agencies will document progress in developing and publishing
an Annual Evaluation Plan:
44 U.S.C. § 3504(e).
The significance of an evaluation study should be defined by each agency and take into consideration factors such as the
importance of a program or funding stream to the agency mission; the size of the program in terms of funding or people served;
and the extent to which the study will fill an important knowledge gap regarding the program, population(s) served, or the
issue(s) that the program was designed to address. Agencies should clearly state their criteria for designating evaluations as
"significant" in their plan. See 5 U.S.C. § 312(b).
20
21
10
•
•
FY 2021 Budget Submission (September 2019): Concurrent with their FY 2021
Budget submissions in September 2019, agencies must document their progress in
creating an Annual Evaluation Plan in the Evidence Template that agencies are
required to complete per 0MB Circular No. A-11.
FY 2022 Annual Performance Plan Submission (September 2020): Concurrent with
the submission of their FY 2022 Annual Performance Plan in September 2020 (to
coincide with the submission of the FY 2022 Budget), agencies must submit an
Evaluation Plan that covers evaluation activities planned for FY 2022. Following this
submission, agencies are expected to provide an Annual Evaluation Plan concurrent
with the submission of their Annual Performance Plan to 0MB in September of each
year, which will be published along with the performance plans the following
February.
(b) Capacity Assessment
CFO Act agencies should immediately begin to plan how they will assess the coverage,
quality, methods, effectiveness, and independence of their statistics, evaluation, research, and
analysis efforts. Non-CFO Act agencies, as well as sub-agencies, operational divisions, and
bureaus of CFO Act agencies are strongly encouraged to plan and conduct a capacity
assessment as appropriate. This assessment should provide a comprehensive view of agency
capacity, while also ensuring that the review takes into account specific needs identified
through the development of the agency's learning agenda. Below are the key activities for
agencies to document progress in developing and undertaking a capacity assessment:
•
•
•
•
FY 2021 Budget Submission (September 2019): Concurrent with their FY 2021
Budget submissions in September 2019, agencies should propose an approach to
conducting the assessment in the Evidence Template that agencies are required to
complete per 0MB Circular No. A-11.
FY 2022 Annual Performance Plan and Budget Submission (September 2020):
Concurrent with their FY 2022 Annual Performance Plan submission in September
2020 (to coincide with the submission of the FY 2022 Budget), agencies must submit
an interim capacity assessment documenting information gathered to date and
outlining steps the agency intends to carry out to complete their full assessment.
Draft Strategic Plan (May, September, and December 2021): As part of their initial
draft Strategic Plan submitted to 0MB in May 2021, agencies must submit an initial
draft capacity assessment that builds on and revises, as needed, the interim
assessment submitted in September 2020. The full draft capacity assessment must be
submitted in September 2021 with the full draft Strategic Plan. The final draft
capacity assessment must be submitted in December 2021 as part of the agency's
final draft Strategic Plan.
Final Strategic Plan (February 2022): As part of their final Strategic Plan posted
publicly in February 2022, agencies must include thefr updated, final capacity
assessment.
11
(c) Open Data Plan
The Evidence Act requires each agency to develop and maintain an Open Data Plan, 22 which,
in general, shall describe the agency's efforts to make government data open to the public.
The Open Data Plan shall be included in the agency's Strategic Information Resources
Management Plan, which describes how information resource management activities help
accomplish agency missions, including how they support government-wide financial
management priorities, to include better integration of budget, cost, and performance data.
These plans shall be updated annually and made publicly available on the agency's website
not later than five days after each update. 23 OMB's forthcoming Phase 2 Guidance will
provide agencies further guidance necessary to implement the Evidence Act's Open Data
Plan requirement.
The Open Data Plan is one approach through which the Evidence Act strengthens the Federal
government's commitment to open data. Many of the other open data requirements in the
Evidence Act will be familiar to agencies because of existing open data responsibilities
articulated in 0MB Circular No. A-130 (Managing 1,iformation as a Strategic Resource) and
0MB M-13-13 (Open Data Policy-Managing Information as an Asset). While the Evidence
Act, in significant respect, carries forward the existing mechanisms for meeting open data
policy objectives, the forthcoming Phase 2 Guidance will substantively update technical
aspects of operationalizing these mechanisms. Until the Phase 2 Guidance goes into effect,
agencies are reminded to continue meeting their existing open data obligations.
Appendices
Appendix A:
Appendix B:
Appendix C:
Appendix D:
Appendix E:
22
23
Components of Evidence
Further Guidance on Learning Agendas
Further Guidance on Constituting Data Governance and Leadership
Further Guidance on Assessments and Planning
Implementation Resources for Agencies
44 U.S.C. § 3506(b)(2).
See 0MB Circular No. A-130, § 5.a.
12
APPENDIX A: COMPONENTS OF EVIDENCE
Figure A.1 depicts and describes four interdependent components of evidence: foundational fact
finding, policy analysis, program evaluation, and performance measurement. Each of these
components informs and directs the others, and many evidence-building activities may be hard to
categorize because they organically include more than one component. As agencies build and use
evidence, per the Evidence Act and this guidance, they should consider these various types of
evidence.
Figure A. 1: Components of Evidence
Policy Analysis
Analysis of data, such as
general purpose survey or
program-specific data, to
generate and inform policy,
e.g., estimating regulatory
impacts and other relevant
effects
Program Evaluation
System atic ana lysis of a
prog ram, policy, organization ,
or component of these to
assess effectiveness and
efficiency
Performance
Measurement
Foundational Fact Finding
Foundational research and
analysis such as aggregate
indicators, exploratory
studies, descriptive statistics,
and basic research
Ongoing, systematic tracking
of information relevant to
policies, strategies, programs,
projects, goals/objectives,
and/or activities
13
APPENDIX B: FURTHER GUIDANCE ON LEARNING AGENDAS
I.
Purpose
The Evidence Act requires that agencies' strategic plans include a section on evidence building
to be developed in conjunction with the agency's process of updating its strategic plan every four
years. These evidence-building plans are systematic plans for identifying and addressing priority
questions relevant to the programs, policies, and regulations of the agency. Thus, evidencebuilding plans are multi-year learning agendas that provide an evidence-building roadmap to
support effective and efficient agency functioning. Learning agendas offer the opportunity to use
data in service of addressing the key questions an agency wants to answer to improve its
operational and programmatic outcomes and develop appropriate policies and regulations to
support successful mission accomplishment. They identify, prioritize, and establish strategies to
develop evidence to answer important short- and long-term strategic questions (i.e., questions
about how the agency meets its mission(s), including about how programs, policies, and
regulations function both individually and in combination) and operational questions (i.e.,
questions about the agency's operations like human resources, grant-making procedures,
financial systems and tracking, and internal processes). 24 In this context, such strategic and
operational questions to be answered can subsequently be integrated into and reflected in the
agency's Strategic Plan and Annual Performance Plan. When items in the learning agenda are
designed to support planned regulatory actions, the agency should cross reference the entry for
that regulatory action in the current Unified Agenda of Federal Regulatory and Deregulatory
Actions.
Learning agendas should be iterative, flexible, transparent, and tailored to both meet an
individual agency's needs and address agency-specific challenges to developing evidence. They
should highlight recent, existing, and future evidence-building activities; make an agency's
evidence-building plans transparent; and promote interest in and support for the studies,
evaluations, and other evidence that will follow from them. Through the necessary consultations
with relevant stakeholders, learning agendas provide an opportunity to align efforts and promote
interagency collaboration in areas of joint focus or shared populations or goals. For agencies
with emerging evidence-building functions, the learning agenda may emphasize and help
coordinate efforts to design and implement new evidence-building activities. For agencies with
more mature evidence-building efforts that may already have comprehensive research plans,
research roadmaps, enterprise learning agendas, or evaluation strategic plans in place, the
learning agenda may highlight and prioritize existing efforts. As a result, there is no single
approach or format for a learning agenda that will be effective for every agency.
A learning agenda is a systematic way to identify the data agencies intend to collect, use, or
acquire, as well as the methods and analytical approaches to facilitate the use of evidence in
policymaking. It promotes deliberate planning of evidence-building activities, including how to
prioritize limited resources and address potential information gaps that may inhibit an agency's
effective mitigation of risks identified through their enterprise risk management processes.
Leaming agendas relate to, but are distinct from research and development planning for the
While a learning agenda may address an agency's human resources as those resources relate to an identified priority question,
learning agendas do not refer to plans for human capital development.
24
purposes of building the stock ofknowledge. 25 They can also complement an agency's
submission to the Unified Agenda of Federal Regulatory and Deregulatory Actions by clearly
documenting how an agency intends to build and use evidence to support its proposed regulatory
actions.
II.
Establishing and Utilizing a Multi-Year Learning Agenda
Requirements for a Learning Agenda
The Evidence Act requires CFO Act agencies26 to have a learning agenda that covers a four-year
period aligned with the strategic plan and that addresses priority questions (i.e., questions
relevant for programmatic, operational, regulatory, or policy decision-making) across the entire
agency (i.e., the entire Cabinet-level Department). Additionally, 0MB strongly encourages nonCFO Act agencies, as well as sub-agencies, operational divisions, or bureaus of CFO Act
agencies to develop and implement their own learning agendas that tie into and are consistent
with, but are not limited by, the agency-wide agenda.
Management of the Development and Implementation Process
Pursuant to the Evidence Act, the agency's designated Evaluation Officer (see Appendix C) shall
coordinate, develop, and implement the learning agenda. The Evaluation Officer shall play a
leading technical role in the development and implementation of the learning agenda at the
agency level and support efforts to develop learning agendas at the sub-agency, operational
division, or bureau levels, particularly as those agendas relate to the agency-wide plan. Other
senior officials who work on evidence-building activities, such as the Statistical Official and
Chief Data Officer, may also play an important role in the learning agenda process. Agency
leadership should be engaged in the development and implementation of the learning agenda, as
their buy-in and support for determining what high-priority questions should be included, which
stakeholders should inform the process, and which staff and data assets will be needed to fulfill
the learning agenda, are critical to success.
Key Components of the Learning Agenda Process
0MB recognizes that there is not just one way to develop a learning agenda. Agencies should
undertake the learning agenda process in a way that is most appropriate to their particular
agency. To meet the requirements for learning agendas outlined in Section 312 of the Evidence
Act, agencies must include the components described further below in the process. Figure B.1
(below) shows one potential way that agencies could order these components to develop a
learning agenda, though there are other approaches, and agencies may find that a different
process best meets their needs.
Pursuant to 0MB Circular No. A-11, Section 210.13, Research and Development programs "focus on knowledge creation or
its application to the creation of systems, methods, materials, or technologies, such as DOE's Solar Energy and NASA's Solar
System Exploration programs."
26 5 U.S.C. § 312(a).
25
15
Figure B.1: Example Learning Agenda Process
Update
Learning
Agenda
;
a
Engage
Stakeholders
on Agency
Learning
Priorities
I
Use Results
to Inform
Agency
Efforts
\
I
identify
Priority
Questions
Undertake
Learning
Agenda
Activities
•
p
Write the
Learning
Agenda
Engage Stakeholders on Agency Learning Priorities. Agencies should consult with internal
and external stakeholders throughout the process to develop and implement a learning agenda as
learning priorities are identified and evidence-building activities are conducted. The purpose of
stakeholder engagement is to make sure that the learning agenda addresses questions that are
relevant, salient, and meaningful to those with direct interests in the agency' s functions, and that /
the learning that results resonates with stakeholders. Agencies should consult with internal
agency stakeholders who play a role in, have a stake in, or will use the results of the learning
process, including agency staff who oversee the designs, processes, operations, or programs
being discussed in the plan; other evaluation, statistics, analysis, data, enterprise risk
management, and performance units and personnel in the agency; policy staff; regulatory staff;
privacy and information law and policy personnel; and agency leadership. Per the Evidence Act,
agencies shall also consult with the public, State and local governments, and representatives of
non-governmental researchers. 0MB also urges broader consultation with additional
stakeholders, including: the Office of Management and Budget, Federal grant recipients,
Congress, and industry and trade groups. Insofar as an agency's learning agenda describes its
plans to build evidence to address specific questions or fill certain purposes, such as supporting
data-driven, performance and strategic review processes and routines, these agendas also signal
agency priorities externally, including opportunities for collaboration with external stakeholders
to answer some priority questions. The agency' s learning agenda should describe its stakeholder
engagement, including the process and participants.
Agencies should gather input in the manner that best meets their needs, most effectively engages
their specific stakeholders, and leverages existing activities and/or requirements whenever
possible, in accordance with applicable law and policy. Potential models for doing so include:
Requests for Information published in the Federal Register, listening sessions with groups of
16
stakeholders, Technical Working Groups, and one-on-one consultations. 0MB recognizes that
agencies may use different approaches at different points in the process, and that it may not be
feasible to engage all stakeholders for all updates to the learning agenda. While agencies have
the option of soliciting feedback and engaging specific stakeholders on their learning agenda as
part of their consultation process on the Agency Strategic Plan27 , agencies may choose to
conduct separate consultations to account for varied interests of different stakeholder groups as it
relates to these components of the agency' s Strategic Plan.
Identify Priority Questions. Agencies should identify priority questions that, when answered,
will have the biggest impact on agency functioning and performance. Agencies may, but are not
required to, tie their questions to strategic goals and strategic planning. Learning agendas should
include both short- and long-term questions of interest to the agency, as well as mission-strategic
and agency-operational questions (see Table B.1 below), in a balance that emphasizes those of
greatest relevance and priority to the agency. In identifying questions, agencies should move
beyond high-level, broad questions, even at the enterprise level, to those that have sufficient
detail to be answerable and useful.
• I
•
:
I
'
I
I
I
'
Short-Term
Mission Strategic
AgencyOperational
27
Long-Term
negatively predict increased numbers of
participants that complete Agency A's
Program B's sessions?
- How can we prospectively monitor and
evaluate county-level interventions and
initiatives funded by Program C in
Agency D?
- What factors are critical for meeting
Agency J's sub-contracting goals?
- What effect did revised procedures have
on Agency K's ability to hire new staff
within 120 days?
What was the impact of online eligibility
forms vs. paper forms for time to program
enrollment for target population L?
- How do Agency M's policies encourage
grant recipients to implement practices to
support stronger grant performance?
See 0MB Circular No. A-11 , § 230.15 .
17
- Have regulatory actions reduced outcome E in
program area F for programs funded by Agency
C?
- Have capacity-building and increased
opportunities for target population G led to
increased participation of target population Gin
intervention H?
- Based on projected demographic changes and
economic forecasts, how will demand for
Program I likely change, and what new
challenges will these changes pose?
- What regulatory, policy, or process
improvements could be made to help strengthen
the agency's oversight and risk management?
- How can Bureau N in Agency P improve
partnerships with local organizations?
How can Agency Q ensure that its workforce
has the ongoing IT skills needed to meet
Agency R's mission?
In defining these questions, agencies should articulate how they define short- and long-term
(e.g., short-term is less than one year, while long-term is over the next four years), and the period
over which they expect to answer these questions. Learning agendas can give agencies the
opportunity to strategically build evidence via systematic questions and agencies should, to the
extent possible, describe how short- and long-term questions relate to one another. For instance,
an agency may want to understand the impact on participants of a large-scale grant program. In
order to answer this long-term question, the agency may need to answer a series of short-term
questions, like "How do grantees implement the program in individual locations?" or "To what
extent do program participants achieve the desired intermediate-term program outcomes?"
Agencies should consider and include the range of methods and types of evidence that could be
brought to bear to answer the priority questions identified in their learning agenda, including
foundational fact finding, performance measurement, policy analysis, and program evaluation
(see Appendix A). Agencies, to the extent practicable, may also answer questions using existing
evidence, including literature reviews, meta-analyses, and research clearinghouses. Indeed,
existing evidence is critical to identify gaps in knowledge and unmet needs, and learning agendas
should address the current state of evidence. The questions to be answered must drive the
research methods (and not vice versa), and agencies are expected to use the most rigorous'
methods possible that align to the identified questions and are feasible in a given circumstance.
Agencies should, thus, be realistic about what can be measured and what cannot when
prioritizing questions. As appropriate, agencies are encouraged to discuss opportunities for
external entities to support the agency's learning agenda, including by explicitly identifying
questions that are amenable to such collaboration.
Write the Learning Agenda. Written learning agendas should include the feedback gathered
from the agency's stakeholder engagement efforts and use clear, plain language. Consistent with
the Strategic Plan, agencies should make their learning agenda public (e.g., by posting on their
agency website and disseminating to stakeholders), though agencies should determine what is
most appropriate to publish based on their own needs, content, and relevant statutory
requirements. At a minimum, agencies should include the following types of content in their
learning agenda, recognizing that this content does not necessarily need to be organized
according to these headings:
•
•
•
•
•
•
Which Strategic Goals and Objectives that the Learning Agenda will Address;
Priority Questions to be Answered;
Activities that the Agency will Engage in to Address Priority Questions;
Timing of Learning Agenda Activities;
Potential Data, Tools, Methods and Analytic Approaches to be Used to Answer Priority
Questions; and
Anticipated Agency-Specific Challenges and Proposed Solutions to Developing Evidence
to Support Agency Priorities.
18
Undertake Learning Agenda Activities. Agencies should carry out the activities specified in
the learning agenda, including conducting evaluations (internally or extemally), 28 data analyses,
performance activities, syntheses of existing evidence, etc. In doing so, agencies are expected to
abide by relevant 0MB guidance. 29 It will be important for agencies to consider ways to combine
or coordinate related evidence and data-related activities, in accordance with applicable law
governing the collection, use, and disclosure of data and information.
Use Results to Inform Agency Efforts. Critical to the learning agenda process is the actual
learning that results from the evidence-building activities an agency undertakes and how the
agency uses and disseminates that information. Agencies should disseminate results internally to
the appropriate parties for learning, improvement, and decision-making. As agencies consult
with internal and external stakeholders, they should consider how to best time dissemination to
maximize the usefulness of the evidence. In addition, agencies should have a plan for external
publication of findings from program evaluations, research studies, and related analyses per
accepted standards of scientific integrity as noted in the agency's evaluation policy or other
procedures. Beyond dissemination, agencies should also consider how they will use results to
inform agency decisions and activities, including, for example, continuous quality improvement,
strategic planning, and budget formulation. For example, agencies can leverage their internal
strategic review process, a key management routine for ensuring agency implementation
strategies and activities can be updated, to improve strategic goals, objectives, and programmatic
outcomes. Agencies must ensure that the use of the results of leaI'J).ing activities is done in
accordance with applicable law and policies governing the collection, use, and disclosure of data
and information.
Update Learning Agenda. Updating the learning agenda builds and maintains the culture of
learning an agency needs in order to think critically and to examine and systematically improve
programs, processes, regulations, and policies. Agencies must revisit their learning agendas at
least annually and update them as needed to reflect progress toward meeting the agency's
original goals and objectives, shifting agency priorities, changing contexts within which the
agency operates, and emergent needs. Leaming agendas should also be updated to incorporate,
when available, the results of the activities an agency undertakes to answer priority questions.
0MB does not expect that agencies will rewrite or draft a new learning agenda annually. Rather,
the focus should be on creating meaningful updates as appropriate.
28 Significant evaluations undertaken as part of learning agenda implementation should be included in the annual Evaluation
Plans required by the Evidence Act.
29 See Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity oflnformation Disseminated by
Federal Agencies; Office ofMgm't & Budget, Exec. Office of the President, 0MB M-05-03, Final Information Quality Bulletin
for Peer Review (2005), available at
https://www.whitehouse.gov/sites/whitehouse. gov/files/omb/assets/OMB/inforeg/igg oct2002.pdf; 0MB M-14-06; principles
and best practices in the Federal Data Strategy (forthcoming); and future 0MB guidance on program evaluation.
19
APPENDIX C: FURTHER GUIDANCE ON CONSITUTING DATA GOVERNANCE
AND LEADERSHIP
Implementing a Data Governance Body
I.
Purpose
Data governance is the process of setting and enforcing priorities for managing data as a strategic
asset in service of an agency's mission, including its strategic plan and other high-level priorities.
This is accomplished by establishing a data governance structure with sufficient authority over
the management of and access to key data assets, including roles for developing, overseeing, and
coordinating data management policy and resource allocation.
Data governance is foundational to undertaking many activities required by the Evidence Act,
and responsibility is shared among multiple parties. Effective implementation of data governance
therefore requires a data governance body that, at a minimum:
•
•
•
•
Supports agency priorities, such as those identified in strategic plans and learning
agendas, by informing Strategic Information Resources Management Plans for the
agency and identifying resources to implement those priorities;
Coordinates and supports implementation of data management responsibilities with
agency data-management actors to strengthen strategic plans and learning agendas;
Sets agency data policy in a manner that complements, but does not supplant the statutory
authority of established positions; and
Coordinates implementation of the Federal Data Strategy by assessing data maturity,
risks, and capabilities to recommend related data investment priorities.
By September 30, 2019, the head of each agency must establish an agency Data Governance
Body, chaired by the Chief Data Officer, with participation from relevant senior-level staff in
agency business units and data support functions and regular meetings.
II.
People
An enterprise-wide Data Governance Body is only effective when the agency head or delegated
authority sponsors the program and assigns senior staff with responsibility for ensuring that the
data governance program addresses key agency concerns and priorities. In addition to the Chair
(i.e., the Chief Data Officer), an agency's Data Governance Body should include appropriate
senior-level staff and technical experts needed to discuss and set policy on a range of data and
data-related topics.
Agencies are encouraged to constitute their Data Governance Bodies in a way that ensures that
they meet the requirements discussed below for that particular agency. At a minimum, agencies
should include the newly-designated positions from the Evidence Act (i.e., the Chief Data
Officer, Evaluation Officer, and Statistical Official) and senior officials from agency program
areas. In addition, 0MB expects that the following positions have defined roles in the
20
governance process and may be members of the Body: Chief Acquisition Officer, Chief
Financial Officer, Chief Freedom oflnformation Act Officer, Chieflnformation Officer, Chief
Information Security Officer, General Counsel, Performance Improvement Officer, Senior
Agency Official for Geospatial Information, Senior Agency Official for Privacy, and Senior
Agency Official for Records Management. Agencies may also augment their Data Governance
Body with other relevant senior officials, such as Chief Operating Officer and Chief Technology
Officer for permanent or rotational participation in accordance with agency governance rules and
customs. The makeup of an agency's Data Governance Body should be driven by the agency's
needs and structure.
III.
Reporting
All agencies must include their Data Governance Body in their Strategic Information Resources
Management Plans' Governance sections as required by 0MB guidance. 30 Additionally all
agencies must report their Data Governance Body members to 0MB via email at
[email protected] no later than September 30, 2019.
Key Senior Officials
The Evidence Act requires agencies to designate three senior officials - Chief Data Officer,
Evaluation Officer, and Statistical Official - who will oversee the use of data and evidencebuilding activities in agencies. The following outlines OMB's expectations for the designation of
these officials, as well as expectations for how these individuals will work with one another
within agencies. 31
Chief Data Officer
I.
Background
While there are many roles in the Federal Government that relate to data management, over the
last few years, Chief Data Officers (CDOs) have emerged to lead organizational development of
processes to leverage the power of data. CDOs enable data driven decision-making in a variety
of ways, from providing and leveraging centralized agency analytics capacity32 to creating tools
and platforms that enable self-service across their agencies and for the public.33 Successful data
management must account for every stage of the data lifecycle. Among other things, it involves
establishing effective procedures, standards, and controls to ensure quality, accuracy, access, and
protection of data, as well as managing information technology and information security. The
See 0MB Circular No. A-130, § 5.a.
As illustrated in the descriptions of functions and responsibilities for each of these positions, it is OMB 's expectation that these
are full-time positions. Agencies should identify and designate staff accordingly.
32 See Office of Mgm 't & Budget, Exec. Office of the President, 0MB M-19-16, Centralized Mission Support Capabilities for
the Federal Government (2019), available at https://www.whitehouse.gov/wp-content/uploads/20 l 9/04/M-19-16.pdf.
33 Jane M. Wiseman, Ash Ctr. for Democratic Governance & Innovation, Harvard Kennedy Sch. Data-Driven Government: The
Role of Chief Data Officers. (2018), available at https://www.innovations.harvard.edu/sites/default/files/Data-DrivenGovernment. pdf.
30
31
21
Evidence Act formally establishes within agencies the position of the CDO, who, through the
functions described below, and in coordination with other agency leaders, will ensure the
successful execution of the agency's data management responsibilities.
II.
Designation of Chief Data Officers
By July 13, 2019, the head of each agency shall designate a nonpolitical appointee employee in
the agency as a Chief Data Officer to support implementation of the Evidence Act and serve on
the Chief Data Officer Council. Agency heads shall designate this role based on the
qualifications below and place this role in the best position within their agency to serve the
functions listed below.
The Evidence Act does not require this position to be within any particular organizational unit of
the agency or to report to any particular official. However, consistent with the role they play in
the agency, CDOs shall serve in a central leadership position, with visibility into relevant agency
operations, and be positioned highly enough to regularly engage with other agency leadership,
including the head of the agency. Agencies, as part of their Information Resource Management
governance planning activities, shall evaluate the effectiveness of the organizational placement
of roles and the allocation of responsibilities.
The CDO shall have the necessary authority to carry out the responsibilities described below.
Nevertheless, some of these responsibilities are shared with others in the agency. Therefore,
these functions should be carried out in consultation with the Data Governance Body and agency
officials who also may have shared responsibilities under other statutes. The agency head is
responsible for assuring that shared responsibilities are carried out in an effective and efficient
manner.
III.
Qualifications for Chief Data Officers
CDOs must have the necessary skills, knowledge, and expertise to carry out the functions
assigned by law and described below. They should be experts in data management and
governance (including creation, application, and maintenance of data standards) and have a
working knowledge of data collection, analysis, protection, use, and dissemination as it relates to
the agency's mission. 34 Specific qualifications include:
•
•
Demonstrated ability to communicate effectively, build coalitions, and exert influence
with political savvy among stakeholders within large organizations, between
organizations and government entities (e.g., between state and Federal government
bodies), and among data users and providers;
Demonstrated ability to successfully manage organizational culture change;
According to section 202(e) of the Evidence Act, "The Chief Data Officer of an agency shall be designated on the basis of
demonstrated training and experience in data management, governance (including creation, application, and maintenance of data
standards), collection, analysis, protection, use, and dissemination, including with respect to any statistical and related techniques
to protect and de-identify confidential data." 44 U.S.C. § 3520.
34
22
•
•
•
•
•
IV.
Demonstrated ability to facilitate and implement best practices in data curation and
management across the entirety of the data lifecycle;
Demonstrated ability to facilitate improved regulatory outcomes by supporting agency
compliance with Executive Order 12866, Regulatory Planning and Review, and 0MB
Circular No. A-4, Regulatory Analysis;
Demonstrated ability to think strategically and solve complex problems regarding the
application of data to data-driven decision-making;
Demonstrated ability to facilitate complex data analysis to improve operational results,
regulatory activities, and mission outcomes; and
Strong business acumen and familiarity with data science approaches, cloud computing,
cyber security, privacy, confidentiality, data analytics, statistical methods, policy
analysis, information quality standards set out in the Information Quality Act (IQA), 35
record disclosure laws and policy, and the Paperwork Reduction Act (PRA).
Responsibilities for Chief Data Officers
Governance Responsibilities
One of the CDO's most important responsibilities is to convene and coordinate agency-wide data
governance, which is a shared activity among multiple data stewards and supporting officials. In
this regard, the Evidence Act assigns specific functions to the CDO, but it is OMB's expectation
that other agency leaders continue to perform the functions assigned to them by relevant statutes.
The head of the agency retains ultimate responsibility for the success of the agency's data
management activities. To further the necessary collaborative environment, the CDO's
convening and coordination functions shall include: 36
•
•
•
•
•
•
35
36
Leading the agency's Data Governance Body, including facilitating collaborative
activities among the numerous actors with responsibilities and needs for data within the
agency;
Supporting the agency's learning agenda by coordinating data access and management
activities that support evidence building, including evaluation, performance reporting,
and the analysis of regulations;
Coordinating with any official in the agency responsible for using, protecting,
disseminating, and generating data to ensure that the data needs of the agency are met;
Engaging agency employees, the public, and contractors in using public data assets, and
encouraging collaborative approaches on improving data use;
Supporting the Performance Improvement Officer of the agency in identifying and using
data to carry out the functions described in 31 U.S.C. § 1124(a)(2);
Supporting the Evaluation Officer of the agency in obtaining data to carry out the
functions described in 5 U.S.C. § 313(d);
Id. § 3516, note.
Section 202(e) of the Evidence Act delineates functions of the Chief Data Officer. See id. § 3520(c).
23
•
•
•
•
•
Reviewing the impact of the infrastructure of the agency on data asset accessibility and
coordinating with the Chief Information Officer of the agency to improve such
infrastructure to reduce barriers that inhibit data asset accessibility;
Ensuring that, to the extent practicable, the agency maximizes its use of data, including
for the production of evidence, including regulatory analyses, cybersecurity, and the
improvement of agency operations;
Identifying points of contact for roles and responsibilities related to open data use and
implementation;
Serving as the agency liaison to other agencies and 0MB on the best way to use existing '
agency data for statistical purposes. This function includes helping program units provide
data to statistical agencies and units, as required in section 303 of the Evidence Act; 37 and
Complying with any regulation and guidance issued under the newly added subchapter III
of chapter 35 of title 44 of the United States Code, including the acquisition and
maintenance of any required certification and training.
Lifecycle Data Management Responsibilities
The second major area of responsibility for a Chief Data Officer is the agency's lifecycle data
management, including: 38
•
•
•
•
•
•
Ensuring that the agency follows, to the extent practicable, data management best
practices; 39
Carrying out information resources management strategic planning, 40 including leading
the development of the agency's Open Data Plan; 41
Carrying out the requirements of the agency with regard to the information collection and
control of paperwork, including certifying to 0MB the agency's compliance with 5 CFR
1320.9 and 1320.8(b)(3);42
Carrying out information dissemination; 43
Managing the agency's data assets, including the standardization of data format, sharing
of data assets, and publication of data assets in accordance with applicable law; 44 and
Carrying out records management. 45
Delegation ofResponsibilities for Statistical Law
The protection of information collected for statistical purposes is a core duty of the statistical
agencies; the ability of statistical agencies to fulfill this duty is contingent on compliance with all
laws and regulations that they are bound to follow. The CDO must delegate the above
responsibilities to the head of a principal statistical agency or recognized statistical unit to ensure
Id. § 3581.
Id. § 3520(c)(l).
39 Id. § 3520(c)(6).
40 Id. §§ 3506(b), 3520(c)(5).
41 Id. §§ 3506(b)(2), 3511.
42 Id. §§ 3506(c); 3507.
43 Id. § 3506(d).
44 Id. § 3520((c)(3).
45 Id. § 3506(f).
37
38
24
compliance with laws and associated policies specific to statistical agencies, including section
314 of title I and all of title III of the Evidence Act, as well as policies promulgated pursuant to
44 U.S.C. §§ 3504(e) and 3506(e) and the statutes authorizing each of the principal statistical
agencies and each of their statistical programs. Statistical agency or unit heads, to the extent
permissible by law, must consult with the CDOs in carrying out said responsibilities, and CDOs
must defer to them with respect to any data acquired, maintained, or disseminated by the agency
under applicable statistical law. 46
Evaluation Officer
I.
Background
Strengthening the program evaluation function within the Federal government is an important
step in expanding evidence building. The Evaluation Officer shall be responsible for overseeing
the agency's evaluation activities, learning agenda, and capacity assessment, as well as
collaborating with, shaping, and contributing to other evidence-building functions within the
agency. The Evaluation Officer is responsible for providing technical and methodological
leadership to assess, improve, and advise on evaluation activities across the agency. For agencies
that are less mature in their evaluation activities, or for those agencies without additional
evaluation expertise distributed throughout the agency, the Evaluation Officer may also be
responsible for conceptualizing, prioritizing, and designing the agency's evaluation activities.
II.
Designation of Evaluation Officers
By July 13, 2019, the head of each of the 24 Federal agencies covered by the CFO Act must
designate a senior employee of the agency as the Evaluation Officer based on demonstrated
technical expertise and skills in program evaluation methodology and practices and without
regard to political affiliation. 47 Each agency must have one Evaluation Officer to serve in this
role for the agency as a whole (e.g., the entire Cabinet-level department). In addition, where
appropriate, 0MB strongly recommends that agencies consider building capacity in subagencies, operational divisions, or bureaus within the agency that also may have a need for an
Evaluation Officer, with the agency-level official providing expert leadership and coordination
among those officials.
Agencies have the flexibility to designate their Evaluation Officer based on the agency's
organizational structure and particular needs. When designating the Evaluation Officer, agency
heads should consider lines of accountability and the coordination needed to ensure the
Evaluation Officer is aligned and partnered with supporting offices and functions to effectively
perform their roles and responsibilities. Regardless of structure, there must be one Evaluation
Officer at the agency-wide level who facilitates and oversees evaluation activities across the
agency. It is important to note that the Evaluation Officer cannot be a member of the Inspector
46
47
Id. § 3520(d)(3).
All other agencies are encouraged to designate Evaluation Officers and implement this guidance.
25
General's staff or located in the Inspector General's (IG) office. While IGs play a very important
role in agency oversight and accountability, the Evaluation Officer required by the Evidence Act
has a different function.
III.
Qualifications for Evaluation Officers
The Evaluation Officer shall be appointed without regard to political affiliation based on
demonstrated, senior-level technical expertise in evaluation methods and practices and
documented expertise to the disciplines of the agency. This includes advanced education and
training in an evaluation-related field, direct experience in applying evaluation methods to study
Federal programs and/or policies, and knowledge of current analytic techniques. Agencies shall
ensure that they appoint or designate an Evaluation Officer with credibility in the range of
relevant evaluation practices and methods, as well as appropriate expertise in the culture,
disciplines, and policy areas of the agency. The Evaluation Officer must have the standing and
authority to perform the functions listed below. Continuity of their leadership and the ability to
oversee multi-year efforts are likewise important. Evaluation Officers should also have
demonstrated ability to:
•
•
•
•
•
•
Ensure that the agency maintains principles of scientific integrity throughout the
evaluation process, including adherence to professional values and practices when
conducting and using the findings from evaluation to maintain objectivity, clarity, and
reproducibility and to insulate from bias, fabrication, falsification, censorship, or other
threats to integrity;
Ensure that rigorous evaluation is a priority that is not subordinated to other agency
demands;
Communicate effectively, build coalitions, and exert influence with political savvy
among stakeholders within large organizations, between organizations and government
entities (e.g., between state and Federal government bodies), and among data users and
providers;
Think strategically about allocation of evaluation resources and dissemination of
evaluation findings to support agency business and mission needs;
Communicate the findings of evidence-building activities to a range of stakeholder
audiences; and
Successfully manage organizational culture change.
The Evaluation Officer must report to a key agency official so that evaluation activities,
including findings, can be integrated into decision-making processes. The Evaluation Officer
must devote sufficient time to this role to meet the expectations and functions outlined in section
313 of the Evidence Act and reflected in this Memorandum.
26
IV.
Responsibilities for Evaluation Officers
The Evaluation Officer shall have authority and responsibility for providing leadership over the
agency's (or sub-agency's, operational division's, or bureau's) evaluation activities, and will be
expected to perform the following functions:
•
Serve as:
o Agency champion for and educator of agency staff and leaders about evaluation,
including what evaluation is, the value of conducting evaluations, how to discern
high-quality evaluation from other types of analyses, and the importance of
evaluation as a strategic investment;
o Senior advisor to agency leaders on issues of evaluation policy and practice, such as
designing and undertaking evaluations, interpreting results, and integrating evaluation
findings into day-to-day agency operations, management processes, budgeting,
strategic planning, and other decision-making;
o Senior agency contact on evaluation for agency-wide and cross-cutting evaluation
efforts, both with external stakeholders and in coordination with senior officials
responsible for other agency functions, including officials responsible for
implementing privacy policy, the Chief Data Officer, the Chieflnformation Officer,
the Statistical Official, the Performance Improvement Officer, additional evaluation
and analysis units and personnel in the agency, and others as appropriate;
o Participant in the agency's Chief Operating Officer-led efforts to review progress on
Agency Priority Goals and other management priorities to ensure that evidence is
included and used appropriately;
o Member of the agency Data Governance Body; and
o Member of the interagency Evaluation Officer Council.
•
Oversee or conduct:
o Assessment of the coverage, quality, methods, effectiveness, objectivity, scientific
integrity, and balance of the portfolio of evaluations, policy research, and ongoing
evaluation activities of the agency, in consultation with other methodologists, such as
the Statistical Official, when appropriate;
o Improvement of agency capacity to support the development and use of evaluation,
coordinate and increase technical expertise available for evaluation and related
research activities within the agency, and improve the quality of evaluations and
knowledge of evaluation methodology and standards;
o Implementation of the forthcoming 0MB guidance for program evaluation standards
and best practices;
o Establishment and implementation of an agency evaluation policy that affirms the
agency's commitment to conducting rigorous, relevant evaluations and to using
evidence from evaluations to inform policy and practice. The policy will provide the
agency's stakeholders with a clear understanding of the expectations related to key
principles, such as evaluation rigor, relevance, transparency, independence, and
ethics;
o Coordination, development, and implementation of the plans required under section
312 of the Evidence Act:
27
•
Learning Agenda (evidence-building plan) in conjunction with the Agency
Strategic Plan;
•
Annual Evaluation Plan, submitted in conjunction with the Annual
Performance Plan; and
• Capacity Assessment, included as part of the Agency Strategic Plan.
o Development of new or improvement of existing processes to integrate evaluation
findings into agency decision-making and other functions;
o Management of agency-wide evaluation standards and requirements to ensure the
scientific integrity of the agency's evaluation activities; and
o Use and dissemination of evaluation results throughout the agency and to the public,
as appropriate.
Statistical Official
I.
_ Background
Statistical expertise allows organizations to ensure that data are gathered, processed, and curated
so as to produce statistical products with the highest standards of data quality while protecting
confidentiality, privacy, and security. Data quality has multiple dimensions, including credibility
and accuracy, timeliness and relevance to valuable decision-making processes, the objectivity
with which data are produced, and their accessibility to multiple users at an appropriate level of
clarity and detail. Applying statistical expertise involves the maintenance and development of
policies that anticipate and address the needs of data users and providers, the continual
advancement and adoption of statistical techniques to maximize the quality of statistical outputs,
and the delineation of procedures to ensure that these policies and techniques are implemented in
a rigorous and efficient manner.
II.
Designation of Statistical Officials
By July 13, 2019, the head of each of the 24 Federal agencies covered by the CFO Act must
designate a statistical official to advise on statistical policy, techniques, and procedures. 48 Within
Federal agencies that contain one or more principal statistical agencies, the mantle of leadership
to cultivate and apply statistical expertise is worn by those principal statistical agencies.
Statistical agencies have developed and demonstrated a high level of statistical expertise over
many decades as the development and implementation of statistical surveys advanced and was
expanded to incorporate a wide array of non-survey data from administrative, commercial,
sensor-based, and other sources. They continue to lead as the field grows and develops
methodologies to improve the quality and efficiency of statistical data uses. Statistical agency
heads provide oversight and advice to the leadership of their parent agencies to promote and
optimally balance the ideals of data quality and confidentiality. As champions of these ideals,
48 5 U.S.C. § 315(a) ("The head of each agency shall designate the head of any statistical agency or unit within the agency, or in
the case of an agency that does not have a statistical agency or unit, any senior agency official with appropriate expertise, as a
statistical official to advise on statistical policy, techniques, and procedures. Agency officials engaged in statistical activities may
consult with any such statistical official as necessary.").
28
principal statistical agencies are a logical choice to play a broad role in overseeing the curation
and use of their parent agencies' data assets. In agencies that do not contain principal statistical
agencies, a variety of other advisory structures or personnel may play similar roles. The
Evidence Act institutionalizes these roles by requiring each CFO Act agency to designate a
Statistical Official. For agencies that house one of the 13 statistical agencies, the agency head
must designate the head of a statistical agency as the Statistical Official. 49 Where one does not
exist, an agency head should identify as a Statistical Official an individual who demonstrates
high levels of expertise as described in the Qualifications section below. 0MB expects agencies
to designate Statistical Officials based on the qualifications they possess and functions they are
to perform as described in the Responsibilities section below, and that the individuals'
qualifications shall take preference over other considerations. The ability of the Statistical
Official to have continued leadership and the ability to oversee multi-year efforts are likewise
important for agencies to consider as they designate this Official.
The Statistical Official must have the standing and authority to perform the functions listed in the
Responsibilities section below. The Official must have sufficient independence to meet the
requirements of the Evidence Act and to carry out the responsibilities of section 302(a) of the
Evidence Act. so The Statistical Official must devote sufficient time to this role to meet the
expectations and functions outlined in sections 101 and 302 of the Evidence Act51 and reflected
in this Memorandum.
III.
Qualifications for Statistical Officials
Each Statistical Official must have appropriate expertise. This includes:
•
•
•
•
•
Demonstrated ability to plan, implement, and direct large-scale ongoing and new statistical
programs;
Demonstrated experience in research and statistics as evidenced by published papers,
presentations at professional meetings, and national recognition as an authority in a
statistical field;
Demonstrated ability to build and maintain productive relationships with high-level
government policy officials, academics, scholars in research organizations, officials in
international organizations, and stakeholders;
Outstanding communication and analytical capabilities;
Knowledge of Federal privacy and confidentiality laws and regulations as they relate to the
curation of data records, the conduct of data collection activities, and the production of
49See id. The 13 principal statistical agencies are the Bureau of Economic Analysis (BEA) in the U.S. Department of Commerce;
Bureau of Justice Statistics (BJS) in the U.S. Department of Justice; Bureau of Labor Statistics (BLS) in the U.S. Department of
Labor; Bureau of Transportation Statistics (BTS) in the U.S. Department of Transportation; Census Bureau in the U.S.
Department of Commerce in the U.S. Department of Commerce; Economic Research Service (ERS) in the U.S. Department of
Agriculture; Energy Information Administration (EIA) in the U.S. Department of Energy; National Agricultural Statistics Service
(NASS) in the U.S. Department of Agriculture; National Center for Education Statistics (NCES) in the U.S. Department of
Education; National Center for Health Statistics (NCHS) in the U.S. Department of Health and Human Services; National Center
for Science and Engineering Statistics (NCSES) in the National Science Foundation; Office of Research, Evaluation, and
Statistics (ORES) in the Social Security Administration; and Statistics oflncome Division (SOI) in the Internal Revenue Service.
50 44 U.S.C. § 3563(a).
51 5 U.S.C. § 311 et seq.; 44 U.S.C. § 3561 et seq.
29
•
•
IV.
one-time and recurring statistical products;
Broad and comprehensive technical knowledge of statistical principles, theory and
methodology, and their application to issues in the agency's subject area; and
Extensive knowledge of statistical programs and related functions and responsibilities of
entities within the agency, as well as a thorough knowledge of statistical programs of other
government agencies.
Responsibilities of Statistical Officials
The Statistical Official will direct and coordinate statistical policy for the agency and will be
expected to perform the following functions:
•
Serve as the:
o Agency champion for data quality to ensure data relevance (e.g., by validating that
data are appropriate, accurate, objective, accessible, useful, understandable, and
timely), harness existing data (e.g., by identifying data needs and reusing data if
possible), anticipate future uses (e.g., by building interoperability of data from its
inception), and demonstrate responsiveness (e.g., by improving data collection,
analysis, and dissemination with ongoing input from users and stakeholders); 52
o Agency champion for confidentiality protection and appropriate data access for
agency data collected with legal or other restrictions on use or release; 53
o Agency senior advisor for statistical policy, including establishing policies to enable
agencies to produce and disseminate relevant and timely statistical information,
conduct credible and accurate statistical activities, conduct objective statistical
activities, and protect the trust of information providers by ensuring the
confidentiality and exclusive statistical use of their responses; 54
o Agency senior advisor for statistical techniques to ensure the use of techniques that
provide optimal levels of data quality and confidentiality; and
o Agency senior advisor for statistical procedures that implement desired policies and
techniques systematically and efficiently. 55
See Office of Mgm't & Budget, Exec. Office of the President, 0MB M-19-01, Request for Agency Feedback on the Federal
Data Strategy, Federal Data Strategy, Principles 4-7 (201"9), available at https://www.whitehouse.gov/wpcontent/uploads/2018/1 0/M-19-0 l .pdf ("Conscious Design").
53 See id. Federal Data Strategy, Principle 2 ("Exercise Responsibility").
54 Section 302(a) of the Evidence Act requires statistical agencies and units to perform these functions through the application of
policies, best practices and procedures, and requires agency heads to "enable, support, and facilitate" their performance of these
responsibilities. 44 U.S.C. § 3563. Such policies, best practices, and procedures should also be promoted at the agency level.
55 Section IOl(a) of the Evidence Act requires the designated statistical official to "to advise on statistical policy, techniques, and
procedures," and directs that "(a)gency officials engaged in statistical activities may consult with any such statistical official as
necessary." 5 U .S.C. § 314( a). Section IO I (a) further requires that "Each statistical official designated under subsection ( a) shall
serve as a member of the Interagency Council on Statistical Policy established under section 3504(e)(8) of title 44. " 5 U.S.C.
§ 314(b).
52
30
•
Oversee:
o Development of agency data quality standards to ensure that quality standards
described in the Information Quality Act and Paperwork Reduction Act are met for
statistical products produced by any part of the agency, including the provision of
metadata to meet stakeholder needs and the permission to use privacy-enhancing
technologies whose use requires significant amounts of documentation; and
o Development of agency confidentiality standards to appropriately safeguard
sensitive information contained in the agency's data assets.
The Statistical Official is expected to play a significant role in the agency's data governance
activities, such as serving on the Data Governance Body. The Statistical Official is also expected
to collaborate and consult regularly with other senior leaders such as the CDO 56 and Evaluation
Officer on issues such as statistical methodologies, data access, data acquisition and linkage,
protecting confidential data, and providing input to the learning agenda.
According to section lOl(a) of the Evidence Act, "(e)ach statistical official ... shall serve as a
member of the lnteragency Council on Statistical Policy" (ICSP). 57 The ICSP, which is chaired
by the Chief Statistician of the United States and coordinated within OMB's Office of
Information and Regulatory Affairs (OIRA), will include the heads of each of the 13 principal.
statistical agencies, as well as the Statistical Official designated by the other CFO Act agency
heads pursuant to the Evidence Act. 58
56 Section 202( e) of the Evidence Act specifies the functional relationship of the Statistical Official with the Chief Data Officer.
See 44 U.S.C. § 3520(d)(l) ("To the extent necessary to comply with statistical laws, the Chief Data Officer of an agency shall
delegate any responsibility under subsection (c) to the head ofa statistical agency or unit (as defined in section 3561) within the
agency."); id. § 3520(d)(2) ("To the extent permissible under law, the individual to whom a responsibility has been delegated
under paragraph (I) shall consult with the Chief Data Officer of the agency in carrying out such responsibility."); id. § 3520(d)(3)
("The Chief Data Officer of the agency shall defer to the individual to whom a responsibility has been delegated under paragraph
(I) regarding the necessary delegation of such responsibility with respect to any data acquired, maintained, or disseminated by
the agency under applicable statistical law.").
57 5 U.S.C. § 314(b).
58 See supra note 43 for a listing of the 13 principal statistical agencies.
31
APPENDIX D: FURTHER GUIDANCE ON ASSESSMENTS AND PLANNING
Annual Evaluation Plan
I.
Purpose
The Evidence Act requires agencies to develop an Annual Evaluation Plan to be submitted in
conjunction with the agency's Annual Performance Plan. These evaluation plans, per the
Evidence Act, describe the evaluation activities the agency plans to conduct in the fiscal year
following the year in which the performance plans are submitted and should include
"significant" evaluations related to the learning agenda and any other "significant" evaluation,
such as those required by statute. 59 The significance of an evaluation study should be defined by
each agency and take into consideration factors such as the importance of a program or funding
stream to the agency mission; the size of the program in terms of funding or people served; and
the extent to which the study will fill an important knowledge gap regarding the program,
population(s) served, or the issue(s) that the program was designed to address. Agencies should
clearly state their criteria for designating evaluations as "significant" in their plans.
Evaluation plans offer agencies the opportunity to plan and document their approach to
evaluation and, in particular, how their intended evaluations will support the agency's learning
agenda. These plans will describe the systematic collection and analysis of information about the
characteristics and outcomes of programs, projects, and processes as a basis for judgments, to
improve effectiveness, and/or inform decision-makers about current and future activities. Annual
Evaluation Plans support effective and efficient government by requiring agencies to think
proactively and methodically about how they will use evaluation to help them operationalize and
implement the activities needed to answer the agency's key questions as laid out in the learning
agenda. Further, the publication of these plans supports more effective government as it allows
the public to understand what an agency plans to evaluate and how it plans to do so. Publication
also holds the agency accountable to undertake those activities. As evaluation is only one of
many types of evidence that agencies will use to answer the priority questions identified in their
learning agendas, the Annual Evaluation Plan is meant to help agencies chart their evaluation
activities, and thus should reflect an agency's needs, priorities, and structure.
A number of agencies already use evaluation plans or comparable documents internally as a way
to organize and monitor their evaluation activities. However, few agencies publish these plans to
the extent required in Section 101 of the Evidence Act. 60 The Evidence Act, for the first time,
requires agencies to create these documents consistently and to publish them. Importantly, this
requirement to publish Annual Evaluation Plan will foster greater accountability and
transparency in Federal evaluation activities and, ideally, the results of those activities. Having a
written evaluation plan will foster openness and encourage agreement with and support of the
purpose, use, and users of the evaluation results. 0MB recognizes that each agency is different,
59 Per section 101 (a) of the Evidence Act, "[t]he term 'evaluation' means an assessment using systematic data collection and
analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency." 5 U.S.C.
§ 311(3).
60 5 U.S.C. § 312(b).
32
and that there will be no single approach or structure for the Annual Evaluation Plan that will
work for all agencies.
II.
Creating an Annual Evaluation Plan
Requirements for an Annual Evaluation Plan
The Evidence Act requires CFO Act agencies to issue an Annual Evaluation Plan concurrent
with the Annual Performance Plan. Additionally, 0MB strongly encourages non-CFO Act
agencies, as well as sub-agencies, operational divisions, bureaus within CFO Act agencies to
develop and implement their own evaluation plans that tie into and are consistent with, but not
limited by, the agency-wide plan as appropriate.
Management and Leadership of the Process
Per the Evidence Act, the agency's designated Evaluation Officer shall lead, coordinate, develop,
and implement the Annual Evaluation Plan. The Evaluation Officer shall play a leading technical
role in the development and implementation of the Annual Evaluation Plan at the agency level
and also support efforts to develop evaluation plans at the sub-agency, operational division, or
bureau level, particularly as the activities in those plans relate to the agency-wide evaluation plan
and learning agenda. Depending on the agency, other senior leaders may play an important role
in creating the Annual Evaluation Plan. Agency leaders are also critical partners in developing
these plans, as their support and buy-in to planned evaluation activities are essential for their
success.
Stakeholder Engagement
Agencies should consult with internal and external stakeholders as they develop and implement
their initial Annual Evaluation Plan and those that will follow. This engagement will likely differ
by agency depending on its needs, stakeholders, evaluation activities, etc. Internal consultation
within the agency should, at a minimum, include those offices and staff that have a role in either
undertaking evaluations or using their results. 0MB also encourages broader consultation with
additional stakeholders to the extent feasible and appropriate. 0MB recognizes that extensive
consultation on an annual basis may be challenging, and urges agencies to undertake processes to
engage stakeholders as effectively and efficiently as possible. The process of stakeholder
consultation, particularly within an agency, on evaluation plans supports collaboration and a
sense of shared purpose, which increases the utility of evaluation findings to improve program
effectiveness for the people and communities served.
Timing ofActivities to be Included in the Plan
The Annual Evaluation Plan will describe the evaluation activities the agency intends to conduct
based on the learning agenda (discussed further in Appendix B) and other statutory mandates
during the fiscal year following the year that the plan is submitted (i.e., for plans that outline
evaluation activities for FY 2022, agencies will submit draft plans to 0MB in FY 2020 with
33
posting of final Annual Evaluation Plans in February 2021). Following the initial submission of
the Annual Evaluation Plan, agencies can use future submissions to update and revise previously
planned evaluation activities based on work conducted and new developments.
Content of the Annual Evaluation Plan
Agencies are encouraged to create Annual Evaluation Plans that reflects the agency's needs and
activities. The plan should not be a laundry list of agency evaluation activities. Rather, it should
explain how, taken together, the planned evaluation activities help to answer the priority
evaluation questions identified in the learning agenda, and either build upon or lay the
foundation for other evidence-building activities: At a minimum, an agency's Annual Evaluation
Plan shall describe the elements outlined below.
Questions to Be Answered. Annual Evaluation Plans shall describe the key questions to be
addressed by each significant evaluation study or phase of an existing evaluation study that the
agency plans to begin in the next fiscal year (i.e., the time period covered by the Annual
Evaluation Plan). 61 In discussing key questions, evaluation plans may necessarily need to
describe a program's purpose, goals, and objectives, as well as discuss how program activities
are linked to their intended effects. As appropriate, agencies may consider including program
logic models in their evaluation plans to visually depict how a program being evaluated is
expected to function and achieve its goals, specifying the program's inputs, activities, outputs,
and outcomes. In this section, agencies should also discuss any evaluation activities that relate to
its proposed regulatory actions in the Unified Agenda of Federal Regulatory and Deregulatory
Actions, recognizing that these activities often need to occur well before the development of
economically significant regulatory actions. To the extent that earlier evaluation activities
influence the activities proposed in the evaluation plan, those should also be mentioned (e.g.,
revisions to evaluation questions based on a review of preliminary findings).
Information Needed for Evaluations. Annual Evaluation Plans shall describe the key
information that will be used in the evaluations outlined in the plan. Specifically, agencies shall
note whether they will undertake new information collections as part of the evaluation studies
discussed or if they will acquire existing information for those evaluations.
Methods to Be Used. Annual Evaluation Plans shall describe the methods that agencies. intend
to use for the evaluations. To the extent practicable, agencies shall articulate the evaluation
design (e.g., experimental design or randomized controlled trial, quasi-experimental design, pre:.
post design, implementation study). As appropriate, agencies may also discuss any other analytic
considerations, such as planned subgroups of interest. 0MB recognizes that providing this level
of detail in advance of the actual initiation of evaluation activities may pose challenges for some
agencies, particularly if they use independent, third-party organizations to conduct program
evaluations and rely on those organizations to provide expert opinions on evaluation designs.
Nevertheless, it is OMB's expectation that in undertaking evaluations, agencies will use the most
The significance of an evaluation study should be defined by each agency and take into consideration factors such as the
importance of a program or funding stream to the agency mission, the size of the program in terms of funding or people served,
the extent to which the study will fill an important knowledge gap regarding the program, population(s) served, or the issue(s)
that the program was designed to address. Agencies should clearly state their criteria for designating evaluations as "significant"
in their plan.
61
34
appropriate method for the research question being asked. To that end, agencies may include
methods that are most appropriate to answer the key questions included in evaluation plans, with
the expectation that agencies will update any methods in future iterations of the Annual
Evaluation Plan.
Anticipated Challenges. To the extent feasible and appropriate, agencies should discuss
anticipated challenges posed by the evaluations included in the Annual Evaluation Plan. For
instance, agencies may anticipate that following participants for several years to measure the
long-term impacts of a program or intervention will be challenging, particularly if the population
being studied is mobile. In this case, the agency might discuss ways to address sample
attenuation. Conversely, agencies may encounter unexpected challenges that necessitate
revisions to previously submitted evaluation plans.
Dissemination. Central to an agency's plans for evaluation activities is how the agency proposes
to disseminate and use the results of those activities. The Evidence Act is clear that agencies
should enhance their capacity to build and use evidence. Thus, agencies should think about and
plan for how they will use evaluation results in making decisions about the programs, policies,
and regulations of the agency before they begin evaluations and not as an afterthought. To the
extent feasible, agencies' Annual Evaluation Plans should include discussion of how they intend
to disseminate and use results for program, policy, and regulatory decision-making, in
accordance with applicable law and policies governing the collection, use, and disclosure of data
and information.
0MB recognizes that forecasting evaluation activities in advance, including the information
described above, may be challenging. In their Annual Evaluation Plans, agencies should address
the above categories of information in the level of detail that is feasible and appropriate given
other considerations, such as availability of funding, procurement processes, data needs, and
related matters. Agencies should be mindful of other laws relating to these areas, including the
Antideficiency Act, the Federal Acquisition Regulation, agency-specific acquisition regulations,
and any other applicable laws. .
As agency learning agendas are expected to include a range of methods and types of evidence
(e.g., foundational fact finding, performance measurement, policy analysis, and program
evaluation) that could be brought to bear to answer the priority questions identified, it is
understood that evaluation will only be one of those methods and be relevant for answering only
some of the identified questions. The Evidence Act requires agencies to provide information on
significant evaluation activities planned. At a minimum, it is OMB's expectation that agencies
will include program evaluation activities in their Annual Evaluation Plans. However, to the
extent that agencies would like to provide context for their evaluation plan by describing other
evidence-building activities they plan to undertake, they are encouraged to do so.
Capacity Assessment
The requirement to carry out a capacity assessment applies to the 24 Federal agencies covered by
th~ CFO Act. Non-CFO Act agencies, as well as sub-agencies, operational divisions, bureaus
within CFO Act agencies are strongly encouraged to carry out a capacity assessment as
35
appropriate. The capacity assessment, per the Evidence Act, requires that agencies provide a list
of existing activities (e.g., programs, initiatives, etc.) and operations (e.g., administrative and
support tasks) that are being evaluated or analyzed. 62 Led by the Evaluation Officer, it must also
include a determination of the extent to which the agency's evaluations, research and analysis
efforts, and related activities are appropriate in terms of:
•
•
•
Supporting agency needs;
Balancing agency needs for learning and management, performance and strategic
management, interagency and private sector coordination, and oversight and
accountability; and
Using appropriate methodologies.
Finally, the Capacity Assessment must determine current agency capacity for:
•
•
Planning and implementing evaluation activities, disseminating best practices and
findings, and incorporating employee views and feedback; and
Carrying out capacity-building activities in order to use evaluation research and analysis
approaches and data in the day-to-day operations.
The Capacity Assessment will provide senior officials with information needed to fulfill the
Evidence Act's intent to improve the agency's ability to support the development and use of
evaluation, coordinate and increase technical expertise available for evaluation and related
research activities within the agency, and improve the quality of evaluations and knowledge of
evaluation methodology and standards.
Agencies are encouraged to conduct their Capacity Assessment using a format and process best
suited to their agency context but which produces an understanding of their agency's strengths
and weakness and promotes the planning for and development of effective capacity-building
activities. The Capacity Assessment is expected to provide agencies with a baseline against
which they can measure improvements to the coverage, quality, methods, effectiveness, and
independence of their agency statistics, evaluation, research, and analyses. 0MB will provide
additional information on the content of this assessment in future communications.
62
5 U.S.C. § 306(a)(8)
36
APPENDIX E: IMPLEMENTATION RESOURCES FOR AGENCIES
0MB recognizes that agencies may need additional resources and support to implement the
Evidence Act requirements addressed in this guidance. Below are several resources that agencies
are encouraged to consult and use as they implement these requirements.
Designated Official Orientation and Ongoing Training
As noted in the Memo, all designated officials (i.e., Chief Data Officers, Evaluation Officers, and
Statistical Officials) must attend an Evidence Act orientation hosted by 0MB in September
2019 focused on how to undertake and operationalize these positions. It is OMB's expectation
that this orientation will provide concrete tools for these designees, as well as an opportunity to
build a network for peer learning.
Federal Data Strategy Tools
The Federal Data Strategy is currently developing a series of tools that agencies can use to
support a range of data-related activities. As they become available, these tools will be available
at https://resources.data.gov.
Evidence and Evaluation Community of Practice
The 0MB Evidence and Evaluation Community page on MAX (available at
https://community.max.gov/x/iA OJQ) includes a curated library of materials to support
agencies in developing learning agendas, writing annual evaluation plans, conducting capacity
assessments, and general program evaluation activities, including:
•
•
•
•
Example agency documents (e.g., learning agendas, evaluation plans, assessments);
Toolkits and other reference tools;
Training opportunities; and
Other materials related to Evidence Act implementation.
37
| File Type | application/pdf |
| File Title | MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES M-19-23 |
| Subject | Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning |
| Author | Russell Vought |
| File Modified | 2019-07-10 |
| File Created | 2019-07-10 |