Maternal, Infant, and Early Childhood Home Visiting Program
Pay for Outcomes Supplemental Information Request (SIR)
Section 3: Requirements Applicable to a PFO Initiative 5
Outcome Measure(s) and Payments 7
Rigorous Third-Party Evaluation 10
No Reduction in Funding for Services 11
PFO Statutory Period of Availability 11
Section 4: Instructions for Responding to the PFO SIR 15
I. Introduction and Project Summary 15
II. PFO Feasibility Study Summary 15
III. Target Population and Evidence-Based Models 16
IV. Stakeholders and Partnership Structure 17
V. PFO Budget and Budget Narrative 17
VI. Selected Outcome Measure(s) 18
VII. Outcome Payments Criteria and Timing 18
VIII. Third-Party Evaluation 19
IX. Verifications and Assurances 22
Section 5: Required Attachments 22
Attachment A – PFO Workplan Timeline 23
Attachment B – Completed Feasibility Study 23
Attachment C – Partnership Agreements with Key PFO Partners 23
Section 6: Submission Information 24
Section 7: Review and Approval Process 24
Appendix A – Instructions for PFO Feasibility Studies 25
Appendix B – Glossary of Selected Terms 28
Through this Supplemental Information Request (SIR), the Health Resources and Services Administration (HRSA) provides state, jurisdiction, and nonprofit Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program recipient with instructions for how they may use a portion of their MIECHV grant funding in relation to a Pay for Outcomes (PFO) initiative.
The MIECHV Program is authorized by Social Security Act, Title V, § 511(c) (42 U.S.C. § 711). Awards to states and nonprofit organizations for the delivery of MIECHV services are authorized under Section 511(c). Section 50605 of the Bipartisan Budget Act (BBA) of 2018 (Pub. L. 115- 123) added new Section 511(c)(3), which authorizes the option for MIECHV recipients to use up to 25% of MIECHV funding for “outcomes or success payments related to a PFO initiative that will not result in reduction of funding for home visiting services delivered by the entity under a childhood home visitation program under this section while the eligible entity develops or operates such an initiative.” The new authority establishes new requirements, including that under Section 511(j)(3)(A), funds used by recipients for a PFO initiative remain available for expenditure by the eligible entity for not more than 10 years after the funds are made available.
The term “pay for outcomes initiative” is defined in newly added Section 511(k) to mean “a performance-based grant, contract, cooperative agreement, or other agreement awarded by a public entity in which a commitment is made to pay for improved outcomes achieved as a result of the intervention that result in social benefit and direct cost savings or cost avoidance to the public sector. Such an initiative shall include—
A feasibility study that describes how the proposed intervention is based on evidence of effectiveness;
A rigorous, third-party evaluation that uses experimental or quasi-experimental design or other research methodologies that allow for the strongest possible causal inferences to determine whether the initiative has met its proposed outcomes as a result of the intervention;
An annual, publicly available report on the progress of the initiative; and
A requirement that payments are made to the recipient of a grant, contract, or cooperative agreement only when agreed upon outcomes are achieved, except for payments to a third party conducting the evaluation.
This SIR provides detailed instructions to MIECHV recipients interested in implementing a PFO initiative funded in whole or in part with MIECHV funds. MIECHV recipients that wish to use MIECHV funds to implement a PFO initiative can submit a notice of proposal to use MIECHV funds for a PFO initiative beginning with the FY 2021 MIECHV Formula Notice of Funding Opportunity (NOFO). They may also do so in future MIECHV formula award applications, subject to availability of funds and to the extent permitted by the authorizing statute and other applicable requirements.
Following a recipient’s submission of a PFO notice of proposal to use MIECHV funds for PFO through an approved MIECHV formula funding application, the recipient must submit a detailed application that responds to this SIR (this application is called the PFO SIR Response) and provides the information required by statute no later than 120 days after the start of the formula funding project period.1 Recipients that do not comply with this requirement may not use funds for a PFO initiative and will be notified that their notice of proposal is being disapproved and that they must submit a revised budget that does not include reservation of funds for PFO outcomes or success payments.
A PFO SIR Response is only required for recipients seeking to budget MIECHV funds for PFO initiative outcome or success payments or PFO initiative evaluation.2 See Exhibit A: MIECHV Funding Application and SIR Response Submission Process below.
Exhibit A: MIECHV Funding
Application and SIR Response Submission Process
The MIECHV Program is authorized by Social Security Act, Title V, § 511(c) (42 U.S.C. § 711(c)) to support voluntary, evidence-based home visiting services for pregnant women and parents with young children up to kindergarten entry living in at-risk communities.3 Decades of scientific research have shown that home visits by a nurse, social worker, early childhood educator, or other trained professional during pregnancy and in the first years of a child’s life improve the lives of children and families. Home visiting helps prevent child abuse and neglect, supports positive parenting, improves maternal and child health, and promotes child development and school readiness.
Evidence-based home visiting can be cost-effective in the long term, with the most significant benefits identified as reduced spending on government programs and increased individual earnings.4 Costs for evidence-based home visiting programs are incurred at the time of service provision, yet participating parents and children may benefit over the course of their lives, and the economic value of improved outcomes typically exceeds costs over the long-term.5 For instance, evidence-based home visiting programs may reduce government spending on special education (Individuals with Disabilities Education Act, Parts C and B), and reduce the need for public assistance programs such as Temporary Assistance for Needy Families (TANF), the Supplemental Nutrition Assistance Program (SNAP), and Medicaid.6 Further, improved outcomes for children and families can also produce social benefit more broadly through the life-long gains associated with better birth outcomes and maternal health, reductions in child abuse and neglect, and increases in family self-sufficiency.
Given the potential to connect evidence-based home visiting program impacts to cost savings and cost avoidance, PFO initiatives represent an innovative approach to funding home visiting services. HRSA anticipates that implementation of a PFO initiative may result in program improvements by MIECHV recipients, including:
Improving service delivery and performance through a focus on targeted outcomes;
Strategically targeting and investing in new or underserved populations;
Expanding service delivery to meet unmet needs by leveraging investors or other funding streams;
Improving data capacity, collection, reporting and management, particularly regarding program performance and achievement of outcomes;
Strengthening stakeholder engagement with families, communities, service providers, other government entities, evaluators, intermediary organizations, and experts with data capacity; and
Improving internal government processes and procurement through an emphasis on improved performance management, data capacity, and rigorous evaluation.
This SIR is intended to complement instructions for how to apply to use MIECHV funds for PFO initiative, which is contained in the FY 2021 MIECHV Formula Notice of Funding Opportunity (NOFO) and may be included in future MIECHV Formula NOFOs or Non-Competing Continuation (NCC) Update guidance. Following notice of a proposal to use MIECHV funds for outcomes or success payments related to a PFO initiative through a MIECHV formula funding application, a response to this SIR (PFO SIR Response) must be submitted to HRSA no later than 120 days after the start of the project period.7
HRSA intends to use the information submitted in response to this SIR to evaluate proposals, provide technical assistance and support to recipients interested in implementing PFO initiatives, and ensure that recipients are proposing PFO projects that meet the statutory requirements.
As noted above, MIECHV recipients have the option to use up to 25% of the grant for “outcomes or success payments related to a PFO initiative that will not result in a reduction of funding for services delivered by the entity under a childhood home visitation program.’’8
A PFO initiative is defined as “a performance-based grant, contract, cooperative agreement, or other agreement awarded by a public entity in which a commitment is made to pay for improved outcomes achieved as a result of the intervention that result in social benefit and direct cost savings or cost avoidance to the public sector.” Such an initiative shall include—
A feasibility study that describes how the proposed intervention is based on evidence of effectiveness;
A rigorous, third-party evaluation that uses experimental or quasi-experimental design or other research methodologies that allow for the strongest possible causal inferences to determine whether the initiative has met its proposed outcomes as a result of the intervention;
An annual, publicly available report on the progress of the initiative; and
A requirement that payments are made to the recipient of a grant, contract, or cooperative agreement only when agreed upon outcomes are achieved, except for payments to a third party conducting the evaluation.9
This section of the SIR provides further information on the requirements associated with a PFO initiative. Instructions for how to demonstrate compliance with these required elements through the submission of the PFO SIR Response can be found in Section 4: Instructions for Responding to the PFO SIR.
Note: MIECHV funds applied to PFO initiatives are subject to all MIECHV statutory and program requirements. In addition to the authorizing statute, please refer to the most recent MIECHV NOFO or NCC Update for information regarding program requirements.
As part of a PFO initiative, MIECHV statute requires the completion of a feasibility study that describes how the proposed intervention is based on evidence of effectiveness.10 The PFO feasibility study assesses the likelihood of a PFO initiative’s success, and provides critical information for deciding whether to launch a PFO initiative. To support objective and successful planning and implementation of a PFO initiative, HRSA recommends that the feasibility study be conducted by an independent third party.
A PFO feasibility study must be completed prior to submitting a MIECHV funding application proposing to use MIECHV funds for outcome payments and evaluation related to a PFO initiative, and prior to the submission of the PFO SIR Response. You may fulfill the PFO feasibility study requirement in one of two ways:
Complete a new MIECHV PFO feasibility study based on the PFO feasibility study instructions, which can be found in Appendix A. You can apply to use MIECHV formula funds to conduct a new PFO feasibility study beginning in FY 2021 through your MIECHV formula award application and in subsequent funding years (contingent on availability of future MIECHV funds). Detailed instructions for what must be included in a MIECHV-funded PFO feasibility study and how to apply can be found in Appendix A; or
Use and submit a feasibility study completed within the past five years that assessed the same intervention and target population you are proposing in the PFO SIR Response. This feasibility study, which may have been supported by non-MIECHV funding sources, can be supplemented with any additional information necessary to submit a complete response to Section 4: PFO Feasibility Study Summary.
MIECHV statute stipulates that MIECHV recipients may choose to allocate up to 25% of the MIECHV award for outcomes or success payments related to a PFO initiative.11 This provision applies only to outcome payments, and does not apply to MIECHV funds used to support other aspects of a PFO initiative (such as a feasibility study, PFO evaluation, reporting on a PFO initiative, data systems, etc.).
An outcome is a positive impact on a target population that you expect to achieve as a result of the intervention over the duration of a project. That outcome is measured by indicators that are observable during the project period.
MIECHV statute further requires that in order to qualify for outcome payments under a PFO initiative, the PFO evaluation must demonstrate that the PFO initiative achieved one or more positive outcomes that result in the following:12
Social benefit to the public sector; and
Cost savings or cost avoidance to the public sector.13
You may propose to select one or more outcome measures for your PFO initiative. You may choose impact outcomes, process outcomes, or both, as long as the methodology for evaluating whether or not the outcome(s) have been achieved allows for the strongest possible causal inference, as required by MIECHV statute.14
To ensure consistency and alignment with the MIECHV program purpose, ALL selected PFO initiative outcome measure(s) must align with the six MIECHV benchmark areas. However, you are not required to address each benchmark area. See the list of MIECHV benchmark areas and constructs in Exhibit B: MIECHV Benchmark Areas and Constructs below.
There are 19 constructs defined by HRSA within the six benchmark areas listed below. At least one PFO initiative outcome measure must align with one of these 19 constructs.
For example, a PFO initiative may choose to focus on improvements in maternal, newborn, and child health, and then further target improvements related to preterm birth, in alignment with construct 1 under MIECHV benchmark area I.
In order to maximize recipient flexibility in structuring a proposed PFO initiative, note that you are not required to operationalize the selected outcome measure the same way that MIECHV requires measurement of the construct for the purposes of annual performance reporting.
Exhibit B: MIECHV Benchmark Areas and Constructs15 |
|||
|
Benchmark Areas |
Constructs |
|
|
Improvements in maternal, newborn, and child health |
|
|
|
Prevention of child injuries, child abuse, neglect, or maltreatment and reductions of emergency room visits |
|
|
|
Improvements in school readiness and child academic achievement |
|
|
|
Reductions in crime or domestic violence |
|
|
|
Improvements in family economic self-sufficiency |
|
|
|
Improvements in the coordination and referrals for other community resources and supports |
|
|
MECHV statute requires that in a PFO initiative, outcome payments are made only when agreed upon outcomes are achieved.16 Setting criteria and timing for outcome payments is an important step in a PFO initiative. You may make outcome payments based on observed differences between the treatment and comparison groups. Alternatively, you may make payments for differential effects for different subgroups. Outcome payments for differential effects might, for instance, entail greater payment amounts for subgroups that are more difficult to serve, have baseline outcomes that are worse than the average for the target population, or are difficult to reach.
MIECHV PFO evaluations, which are further described below, do not require statistical significance in order to trigger an outcome payment. Any observed differences in the intended direction between the treatment and comparison groups may be defined as success, and trigger an outcome payment. Alternatively, you may propose a different approach for setting outcome payment criteria.
Outcome payments can only be made for those outcomes included in the PFO initiative evaluation, and payments can only be made after the evaluator has determined whether the outcomes have been achieved. If applicable, outcome payments may be made for longer-term outcomes that occur after home visiting service delivery ends. For example, a reduction in child maltreatment or emergency room visits might occur as a result of the services provided, but may not be observed until after the home visiting services have been completed. However, all payments must occur within the PFO project period.
MIECHV statute stipulates that funds made available for a PFO initiative will remain available for expenditure for no more than 10 years after the funds are awarded.17 Therefore, the PFO project period cannot be longer than 10 years, but it might be shorter. For example, home visiting services may be performed for 5 years, but an outcome resulting from the program for a final cohort might not be achieved until year 7. Examples of such longer-term outcomes might include reductions in emergency room visits by age two, reductions in child maltreatment, or improvements in kindergarten readiness.
While not required, HRSA encourages outcome payments to be made as soon as evaluation results are available. Payments can only be made after the third-party evaluator has determined that outcome(s) have been achieved.
In accordance with the MIECHV statutory requirements, outcome measure(s) in a PFO initiative must result in social benefit and cost savings or cost avoidance to the public sector.18 When proposing outcome measure(s) and payment amounts, you must provide your rationale for selecting the payment amounts, and describe how the proposed measure(s) and payments take into consideration federal, state, and/or local public sector cost savings, cost avoidance, and/or social benefit. Note that outcome payment amounts are not required to equal amounts of anticipated cost savings, cost avoidance, or social benefit, but your PFO SIR Response should describe how the amounts were determined.
To meet this requirement, you may provide local data, cite the research literature, and/or include a logic model demonstrating how the improvement in an outcome measure results in cost savings, cost avoidance and/or social benefit.
For example, if reduction in preterm birth is chosen as an outcome measure, you may cite research literature or local data as to the anticipated cost savings or cost avoidance associated with a reduction in preterm birth. You may also include a logic model that draws on the literature regarding the longer term impact of reducing preterm birth, such as reduction in developmental delays and longer term health, and how those impacts are likely to result in future cost savings, cost avoidance, and/or social benefit.
MIECHV statute requires that a PFO initiative include a rigorous, third-party evaluation that uses experimental or quasi-experimental design or other research methodologies that allow for the strongest possible causal inferences to determine whether the initiative has met its proposed outcomes as a result of the intervention.19 In addition to the primary purpose of the PFO evaluation, which is to determine whether the outcome(s) in the PFO initiative have been met, the evaluation of the PFO initiative can also contribute to the MIECHV Learning Agenda, build evidence for the home visiting model(s) implemented, and provide critical information for continuous quality improvement (CQI).
The third-party evaluation must be conducted by an independent evaluator with experience conducting rigorous evaluations of program effectiveness including, where available, well-implemented experimental or quasi-experimental methodologies on home visiting or similar interventions. Payment for the evaluation must not be tied in any way to the achievement of the outcomes, and the independent evaluator must not have a financial or other stake in the project that would undermine its objectivity. It is important that the selected evaluator, whose responsibilities include assessing whether the project has achieved the outcomes on which payment by the federal government are conditioned, has the necessary independence from the project to support objectivity.
You will submit a detailed PFO evaluation proposal in the Narrative section of the PFO SIR Response. Instructions can be found in Section 4: Third-Party Evaluation below.
Additionally, in the PFO Budget and Budget Narrative section of the PFO Narrative submission, you will provide detailed information regarding the cost of the PFO evaluation, unless the PFO evaluation is not funded with MIECHV funds. PFO evaluation costs should include salary for staff working on the evaluation, funds for external evaluators, cost of relevant data collection, travel, communication tools, printing, supplies, equipment, participant incentives, etc., and should not include the amount of outcome payments. The budget for evaluation activities should be:
Appropriate for the evaluation design and question(s);
Adequate to ensure quality and rigor; and
Aligned with available program and organizational resources.
See Section 4: PFO Budget and Budget Narrative for more information.
You must demonstrate that dedicating funding for the expenditures associated with the PFO initiative does not result in a reduction of funding for services, as delivered by the MIECHV recipient. MIECHV recipients will be required to make this demonstration by comparing funding levels for service delivery to the year prior to the initiation of the PFO initiative.20 This requirement applies to ALL expenditures associated with a PFO initiative including, but not limited to:
PFO outcome payments;
Costs associated with conducting a PFO feasibility study;
Costs associated with implementation of a PFO evaluation;
PFO reporting costs; and
The portion of implementation and administration costs associated with the PFO initiative, which may include contracts, data capacity expenditures, supplies, travel, equipment, professional development, continuous quality improvement (CQI), technical assistance (TA), and any other costs required to support implementation of a PFO initiative.
Recipients must also consider this requirement in the context of other program and model fidelity requirements, such as maximum caseload capacity and home visitor recruitment and retention, and ensure that all applicable requirements are met.
MIECHV statute stipulates that funds for a PFO initiative within a fiscal year shall remain available for expenditure for up to 10 years after the funds are made available.21 The 10-year period of funding availability (“PFO statutory period of availability”) applies to any MIECHV funds used for a PFO initiative, including outcome payments, evaluation, feasibility study, and other expenses associated with a PFO initiative.
In order to ensure that funding is used to support MIECHV program objectives, taking into account the standard statutory period of availability for use of funds for recipients for non-PFO activities, MIECHV grant funds that are budgeted for a PFO initiative should be rebudgeted to non-PFO activities ) within the first 12 months following the start of the project period. For more information, see Exhibit C: Period of Availability Example.
Note: When planning your PFO initiative, your proposed PFO project period cannot exceed the 10-year PFO statutory period of availability of funds, however you can propose a shorter PFO project period. Your proposed PFO project period should reflect the appropriate amount of time to observe and achieve the outcome measures, complete the PFO evaluation, and ensure that funds are obligated within the PFO statutory period of availability. Any grant funds that have not been obligated for expenditure by the recipient during the period of availability will be deobligated.
Funding for a PFO initiative can be allocated in multiple years as part of a single PFO initiative, however the proposed PFO project period as described in the SIR should reflect the entire PFO initiative, subject to the 10-year PFO statutory period of availability. Only one PFO SIR Response is required for each PFO initiative, but if you propose to budget funds in multiple years as part of a single PFO initiative, you must apply for PFO funds each year through your annual MIECHV formula application. Additional instructions will be provided in the MIECHV Formula Notice of Funding Opportunity/Non-Competing Continuation Update guidance.
Exhibit C: Period of Availability Example |
||
|
Standard Statutory MIECHV Period of Availability
|
PFO Statutory Period of Availability
|
Applicability |
Applies to MIECHV funds allocated for any activities EXCEPT PFO Initiatives. |
Applies ONLY to MIECHV funds PFO funds allocated for A PFO initiative. |
FY 2021 Project Period (Assumes awards made September 30, 2021) |
Standard statutory MIECHV period of availability ends September 29, 2023. |
PFO statutory period of availability ends September 29, 2031.
Note: Funds allocated for a PFO initiative that are subject to the 10-year period of availability should not be rebudgeted to typical MIECHV activities after September 29, 2022. |
MIECHV funds budgeted for a PFO initiative are subject to the standard MIECHV 10% statutory limit on use of funds for administrative expenditures. The 10% administrative expenditure cap22 applies to the total MIECHV Formula Award, including typical MIECHV program administration and any costs associated with PFO administration.
Activity costs associated with a PFO initiative that would typically be subject to the 10% administrative cap (such as reporting costs, subrecipient monitoring expenses, etc.) are subject to the 10% cap. Please refer to the most recent MIECHV NOFO or NCC Update for detailed guidance regarding the types of activities that fall under the administrative expenditure cap.
MIECHV funds budgeted for a PFO initiative are subject to the standard MIECHV 25% limit on recipient-level infrastructure expenditures.23 In other words, costs for PFO initiative activities typically subject to the 25% recipient-level infrastructure limit, such as professional development and training for recipient-level staff, model affiliation and accreditation fees, technical assistance provided by the recipient to the local implementing agencies (LIAs), etc., are also subject to the 25% limit.
The 25% limit on use of funds for recipient-level infrastructure expenditures applies to the total MIECHV Formula Award, and thus includes these expenditures to support standard MIECHV program implementation and PFO implementation.24 Please refer to the most recent MIECHV NOFO or NCC Update for detailed guidance regarding recipient-level infrastructure expenditures.
MIECHV statute requires an annual, publicly available report on the progress of the PFO initiative.25 This annual PFO progress report, which is required for all recipients implementing any component of a PFO initiative, must be made publicly available by posting on your agency/program website or through another public-facing platform. This annual report is due to HRSA no later than 90 days after the end of each federal fiscal year during which a PFO initiative is implemented. Additional instructions for submission of the report will be provided by your HRSA Project Officer no later than 90 days before the PFO annual progress report due date.
For more information on the entire PFO application and reporting process, see Exhibit D: MIECHV PFO Initiative Development, Application, and Reporting Process.
Exhibit
D: MIECHV PFO Initiative Development, Application, and Reporting
Process Complete
a feasibility study to determine if a MIECHV PFO initiative is
feasible. If
you have not yet completed a feasibility study, you can apply to
use MIECHV funds to do so through your MIECHV formula funding
application. You do NOT
need to complete a PFO SIR Response at this time.
If
the PFO initiative is found to be feasible and you choose to apply
to use MIECHV funds for a PFO initiative outcome payments and PFO
evaluation, you may do so beginning with the FY 2021 MIECHV formula
funding application.
Following
review and approval of the MIECHV formula funding application,
complete a PFO SIR Response.
The
PFO SIR Response is due 120 days after the project period start
date.
Use
the findings of the feasibility study to inform the plan described
in the PFO SIR Response. HRSA
will review the PFO SIR Response to ensure alignment with the
MIECHV PFO SIR requirements. PFO initiative implementation can
begin following HRSA approval of the PFO SIR Response.
Submit
Annual PFO Progress Reports Annual
reports on the progress of the PFO initiative are due to HRSA 90
days after the end of the federal fiscal year during which the PFO
initiative is conducted, and must be made publicly available by
the recipient.
A
final progress report will be due to HRSA at the end of the PFO
project period documenting final evaluation results, the extent to
which outcomes were achieved and outcome payments made, and other
key information on the PFO initiative.
This Supplemental Information Request (SIR) provides recipients with instructions for how to meet the statutory requirements associated with implementing a PFO initiative. A complete PFO SIR Response includes:
PFO Narrative
Required Attachments
Attachment A - PFO Workplan Timeline
Attachment B - Completed Feasibility Study
Attachment C - Partnership Agreements with Key PFO Partners
The PFO Narrative must include the following sections:
Introduction and Project Summary
Begin your PFO Narrative with a brief introduction section that describes the purpose, goals, and approach for your PFO initiative. This section should include why you believe PFO is an appropriate strategy to achieve your MIECHV program goals, as well as the specific goals of the PFO initiative. For example, goals for the PFO initiative could include: to expand service delivery to unserved populations; improve birth or other outcomes for the target population; leverage private funding sources for evidence-based home visiting; shift funding from paying for activities to paying for outcomes; funding and/or quantifying the fiscal and social benefits of improving specific outcomes; etc.
Please also include a brief summary of your approach (a high-level project summary), including the PFO project period.
In this section, provide a summary of the results of your PFO feasibility study. This section should address the following:
Was this a new PFO feasibility study, or did you use (and/or modify/supplement) an existing feasibility study that meets MIECHV PFO feasibility study requirements?
If you modified/supplemented an existing study, summarize what information was modified from/added to the original study.
Summarize the key findings from the feasibility study, including:
The conclusion as to whether the PFO initiative is feasible;
The proposed outcome measure(s);
The proposed evidence-based home visiting interventions and the evidence that the intervention is likely to result in improvement in the outcome measure(s) for the target population; and
The capacity of LIAs and service providers to implement a PFO initiative and serve the target population.
Discuss the primary benefits and assets associated with implementation of a PFO initiative, as identified in the feasibility study.
Discuss the primary risks and challenges associated with implementation of a PFO initiative as identified in the feasibility study, and how you propose to mitigate these risks and address challenges in implementation of the PFO initiative.
Discuss how the feasibility study determined that a MIECHV PFO initiative can be implemented and funded without causing any reduction in funding for services.
If your proposed PFO initiative includes new or different activities from those tested in the feasibility study, please note and explain your reasoning for these changes.
You must submit your completed feasibility study as Attachment B. For more information, see Section 3: Feasibility Study Requirements.
In this section, describe the following:
The at-risk community(ies) that will be served through the PFO initiative, based on the findings from your most recent statewide needs assessment;
The LIAs that will participate in the PFO initiative;
The number of families you propose to serve through the PFO initiative;
The total caseload of family slots funded in your most recently approved MIECHV Formula funding application, noting which will be served as part of the PFO initiative, and which will not;
Any potential or anticipated barriers to enrolling eligible families in the PFO initiative, and how you will overcome those barriers;
If you are leveraging non-MIECHV funding streams to partially support the PFO initiative, note the total number of families served through the PFO initiative, as well as how many are MIECHV-funded (please refer to the MIECHV definition of caseload of family slots); and
Which evidence-based home visiting model(s) you propose to implement through the PFO initiative.
PFO initiatives require the involvement of many partners, including LIAs, third-party evaluators, model developers, agencies that house administrative data sources, early childhood systems partners, home visitors, families, and others. They may also include investors and/or an intermediary organization, though neither is required. In this section, describe:
How you are collaborating with external partners on the proposed PFO initiative, as well as any key partners that are internal to your (recipient’s) agency;
The partnership structure, including roles and responsibilities of each partner;
Your plan for communicating about the PFO initiative with home visiting service providers (including home visitors) and the families served; and
Any anticipated challenges or barriers to engaging families in the design and planning of the PFO initiative, and how you will overcome those barriers.
If external funders are supporting the initiative, describe their role, the amount of external funding that is anticipated, and how it will support the PFO initiative.
Note: Attachment C requires partnership agreements with each key PFO partner.26
The goal of the PFO Budget and Budget Narrative is to complement your most recently approved MIECHV budget and budget forms (as documented in your most recently approved MIECHV Formula Award funding application and Notice of Award27) and provide a more detailed description of how each budgeted line item will support the achievement of the PFO initiative’s proposed objectives. This section of the PFO Narrative is not intended to replace your most recently approved MIECHV budget and relevant budget forms.
In this section:
Describe your plans for budgeting MIECHV funds over the entire PFO project period, and state if you plan to allocate MIECHV funds in multiple years as part of a single PFO initiative, and if so, how you will track those funds appropriately.
Provide a line item budget narrative for all program expenses related to PFO using the budget categories in the SF-424A that clearly indicates:
The total amount of funding for outcome payments, including the amount budgeted each year of the PFO project period;
The total amount of funding for PFO evaluation, including the amount budgeted each year of the PFO project period;
Other PFO-related expenditures, such as costs associated with conducting a PFO feasibility study, staff salaries, and data capacity expenditures.
Note: If you plan to allocate MIECHV funds in multiple years as part of a single PFO initiative, this budget narrative should reflect funds that will be budgeted for PFO over the entire PFO project period. If you do NOT plan to set aside MIECHV funds in multiple years, and if there have been no changes to your most recently approved budget, the line item budget narratives can be copied from your most recent MIECHV Formula Award funding application. If there have been changes, please highlight those changes in your PFO SIR Response. Following review and approval of this SIR, you will be required to submit the changes through the regular Prior Approval Request process in HRSA’s Electronic Handbooks (EHBs).
The selected outcome measure(s) for the PFO initiative, including:
How each outcome measure is aligned with the MIECHV benchmark areas; and
How at least one outcome measure aligns with one of the MIECHV performance indicators or systems outcome measures.
Evidence that the selected intervention (evidence-based home visiting model) can be expected to produce the selected outcome(s).
How the outcomes measure(s) would result in meaningful positive outcomes for children, families, and/or communities.
How the proposed outcome measure(s) for the PFO initiative are reflective of federal, state, and/or local public sector cost savings, cost avoidance, and/or social benefit.
Note: You may provide local data, and/or cite the research literature, and/or include a logic model demonstrating how the improvement in an outcome measure results in cost savings, cost avoidance and/or social benefit, including the sources of these.
See Section 3: Outcome Measure(s) and Payments for more information.
In this section, address how you will know when the intervention has resulted in improvement, if/when outcome payments should be made, and outcome payment amounts. When designing outcome payment triggers, recipients and evaluators should work together to determine the criteria for payment. Additionally, you may choose the methodology to determine payment amounts for each outcome measure.
Describe:
How the outcome(s) will be measured;
How you will know when you have achieved the desired outcome;
The payment amounts for each selected outcome measure;
The rationale and method for determining the payment amount per outcome;
How the payment amounts are appropriate and reasonable relative to the proposed outcome measure(s), based on anticipated cost savings or cost avoidance and social benefit;
The payment schedule timeline for outcome payments, stipulating each anticipated payment point; and
To whom the payments will be made (specify if payments will be made to private funders/investors).
Note: A detailed analysis quantifying the federal, state and local public sector cost savings, cost avoidance or social benefit for each outcome is not required as part of the PFO SIR Response.
See Section 3: Outcome Measure(s) and Payments for more information.
In this section, describe the critical elements that will be included in the evaluation of a PFO initiative, as described below.
Specify the entities/organizations responsible for the evaluation. Include the qualifications of individuals designing and overseeing the evaluation and ensuring its quality, including their education or training, and type and years of experience.
Describe how evaluators will have the necessary independence from the project to support objectivity.
Describe the experience of the evaluator in conducting research utilizing the experimental, quasi-experimental, or other research methodology proposed as part of the PFO evaluation, as well as in working with datasets the PFO initiative expects to use.
List all evaluation aims, evaluation questions, and any sub-questions. Ensure that questions are specific, measurable, and clearly listed.
Specify the experimental, quasi-experimental, or other research methodologies that allow for the strongest possible causal inference to determine if the outcome(s) in a PFO initiative have been achieved.
Provide a rationale for the selected evaluation design(s) that includes a clear and well-supported justification for why the proposed design allows for the strongest possible casual inference. For example, how is (are) the selected evaluation design(s) particularly well suited for answering the evaluation questions and, in particular, demonstrate that the outcomes are achieved as a result of the intervention? Were other evaluation designs considered and ultimately not selected due to contextual considerations or limitations?
Discuss how the proposed design meets expectations for rigor described in Exhibit E: Rigor in Evaluation.
Exhibit E: Rigor in Evaluation |
Credibility/Internal Validity: Ensuring what is intended to be evaluated is actually what is being evaluated; ensuring that the method(s) used is the most definitive and compelling approach that is available and feasible for the question being addressed. |
Applicability/External Validity: Generalizability of findings beyond the current project (i.e. when findings “fit” into contexts outside the study situation). Ensuring the population being studied represents one or more of the populations being served by the program. |
Consistency/Reliability: When processes and methods are consistently followed and clearly described so that someone else could replicate the approach and other studies can confirm what is found. |
Neutrality: Producing results that are as objective as possible and acknowledge the bias and limitations brought to the collection, analysis, and interpretation of results. |
Describe the outcome(s) measured in the PFO initiative to be included in the evaluation, in addition to the timeline (by cohort) for when you expect them to occur and when you expect to evaluate whether they have been achieved.
Describe the metrics that will be used to determine if the outcome(s) have been achieved as a result of the intervention (i.e. outcome(s) and outcome targets (if applicable)). This section should expand upon the discussion of outcome(s) included in the outcome payments section, and should discuss:
How the metrics will be measured; and
How the metrics are independent and objective indicators of impact and are not subject to manipulation by the service provider or other project stakeholders, if any.
Explain how the evaluator will collect or gain access to the metrics that will be used.
Describe each data collection method, including the data collection instruments or tools, and explain why the method was selected. For all tools and instruments, include:
Relevant information about the reliability and validity of the measures;
The target population (or sub-population) for each data collection method; and
A demonstration of how the proposed data collection methods (e.g., tools, instruments, respondents) align with the proposed outcome(s).
Be sure to collect data at the frequency with which is needed to ensure outcome payments.
A description of consistent data collection methods and time points across the intervention and control groups (if applicable).
If utilizing administrative data, describe:
The data source(s) (e.g., periodicity, key constructs, target population);
The anticipated quality (e.g., completeness, extent of missing data); and
How the data will be accessed (e.g., publically available, data sharing agreements).
Describe how the sample will be selected to minimize selection bias. Define the type of sampling you chose (e.g., random, stratified, composite, or convenience). How will selection bias be addressed in the evaluation?
Describe the method of participant selection. Describe your sampling recruitment strategy and inclusion/exclusion criteria for determining your sample. How will you account for sampling and measurement error? Describe any possible sampling bias and plans for mediation.
Describe expected attrition rate for study sample and plans for mediation.
PFO evaluations do not typically require statistical significance in order to trigger an outcome payment. Any observed differences between the treatment and comparison group can be defined as success and trigger an outcome payment. Some PFO evaluations do include analysis of statistical significance as part of a learning agenda, however, and recipients may choose to include the determination of statistical significance in the evaluation even if it is not required to trigger an outcome payment.
If there is a plan to determine statistical significance at any point during the evaluation period, provide estimated power to detect impacts.
What were the parameters used to generate the statistics? Report alpha, N, effect size estimates, and the source of the effect size estimates.
Provide a justification that your sample size is appropriate to determine whether outcome(s) have been achieved, and include results from power analysis, if applicable.
For evaluation designs that include a comparison group, include the following:
The method used to allocate participants to groups. For randomized controlled trials (RCTs), for example, how will the randomization method ensure that every participant is as likely as any other participant to be assigned to the treatment (or control) group?
The strategies to address the risk of contamination. What strategies will be implemented that ensured minimal crossover or contamination of members from the comparison group?
How baseline equivalence will be established for key characteristics prior to the intervention, and identification and justification for key characteristics of populations to establish baseline equivalence.
Describe the planned analytic approach for addressing missing data (such as regression imputation, multiple imputation, maximum likelihood, non-response weights).
Describe how the validity and reliability of the analytic methods/results will be established. For example, use of standardized instruments, description of statistical methods used, etc.
Describe the analyses planned for each expected outcome.
Describe potential confounding factors and efforts to control for them.
Acknowledge potential limitations or biases to study design. Are there any study limitations expected? For example, limitations due to sample size, reliability of data, language fluency, etc.
Explain how anticipated limitations can be addressed in advance and/or minimized.
Provide a timeline for all major evaluation phases and activities.
Under the MIECHV statute, the PFO initiative must not result in a reduction of funding for services delivered by the recipient entity administering the MIECHV home visiting program.28 In this section of the PFO Narrative, provide:
An assurance that there will be no reduction in funding for home visiting service delivery as delivered by the MIECHV recipient/recipient, compared to the year prior to the initiation of the PFO initiative,29 as a result of the planning and implementation of the PFO initiative; and
An assurance that there is no anticipated reduction in caseload of family slots compared to the year prior to the initiation of the PFO initiative as a result of this PFO initiative.
A complete PFO SIR Response includes three required attachments:
Attachment A – PFO Workplan Timeline
Attachment B – Completed Feasibility Study
Attachment C – Partnership Agreements with Key PFO Partners
As Attachment A, provide a detailed workplan for implementation of the PFO initiative, based on the PFO Workplan Timeline Template Attachment.
The PFO Workplan Timeline must extend across the entire length of the proposed PFO project period and include start and completion dates for activities. The proposed PFO project period for the PFO initiative should be clearly articulated, and should reflect the appropriate amount of time to observe/achieve the outcome measure(s), complete the PFO evaluation, and ensure that funds are obligated within the PFO statutory period of availability.
Please refer to the PFO Workplan Timeline Template Attachment for additional detail.
As Attachment B, submit the feasibility study (new or modified/supplemented existing study). The PFO feasibility study must be completed prior to proposing to use MIECHV funds for outcome payments related to a PFO initiative and prior submission of the PFO SIR Response.
You may fulfill the PFO feasibility study requirement in one of two ways:
Complete
a new MIECHV PFO feasibility study based on the PFO feasibility
study instructions, which can be found in Appendix
A; or
Use and submit a feasibility study completed within the past five years that assessed the same intervention and target population you are proposing in the PFO SIR Response. This feasibility study, which may have been supported by non-MIECHV funding sources, can be supplemented with any additional information necessary to submit a complete response to Section 4: PFO Feasibility Study Summary.
As Attachment C, include a partnership agreement with each key PFO partner30 that articulates:
Clearly defined roles and responsibilities of each partner;
An assurance that all partners have reviewed the evaluation plan; and
An agreement by all partners to cooperate in the implementation of the evaluation plan, as necessary.
The partnership agreements must either be signed or, if submitted in draft form, must be accompanied by signed letters of intent from each partner to enter into such an agreement.
Following submission of a PFO notice of proposal to use MIECHV funds for PFO through an approved MIECHV Formula funding application, a PFO SIR Response should be submitted to HRSA in response to a Request for Information (RFI) via the HRSA Electronic Handbooks (EHBs) no later than 120 days after the start of the formula funding project period.
Note: A PFO SIR Response is only required for recipients seeking to budget MIECHV funds for PFO initiative outcome or successpayments or a PFO initiative evaluation.
A complete PFO SIR Response submission must include all required sections:
PFO Narrative
Required Attachments
Attachment A - PFO Workplan Timeline
Attachment B - Completed Feasibility Study
Attachment C - Partnership Agreements with Key PFO Partners
HRSA staff will review applications for completeness and compliance with the requirements outlined in this SIR. HRSA staff will either accept the submission as complete and compliant or request additional information or clarification. Recipients may not begin implementation of the proposed PFO initiative until HRSA approves the complete submission in response to the PFO SIR.
A PFO initiative must include a feasibility study, which describes how the proposed intervention is based on evidence of effectiveness. The feasibility study also serves as a tool to determine the viability of using a MIECHV PFO approach to meet the proposed outcome(s), while meeting all MIECHV statutory and program requirements.
The PFO feasibility study must be completed prior to proposing to use MIECHV funds for outcome payments related to a PFO initiative in your MIECHV Formula application, and prior to the submission of the PFO SIR Response. You may fulfill the PFO feasibility study requirement in one of two ways:
Complete a new MIECHV PFO feasibility study based on the PFO feasibility study instructions, below; or
Use and submit a feasibility study completed within the past five years that assessed the same intervention and target population you are proposing in the PFO SIR Response. This feasibility study, which may have been supported by non-MIECHV funding sources, can be supplemented with any additional information necessary to submit a complete response to Section 4: PFO Feasibility Study Summary.
Note: The following instructions are intended to inform and support the development of the feasibility study for those recipients that propose to use MIECHV funds for a PFO feasibility study, and are provided here for additional context and information only. You are NOT required to respond to these instructions in your PFO SIR Response.
When conducting a feasibility study, you should:
Identify and consider what funding source will be used to fund any part of the PFO initiative, in addition to MIECHV funds. You should also consider and ensure that there is sufficient funding for the full term of service provision in the PFO initiative.
PFO initiative funding sources identified in the feasibility study can include:
MIECHV funding;
Provider or LIA working capital;
Foundation funding; and/or
Investor funding.
In your feasibility study, consider the target population for the PFO initiative:
The unmet need for home visiting services that the PFO initiative will address, and the baseline outcome(s) that the PFO initiative seeks to improve;
The at-risk communities the PFO initiative will serve (based on the findings from your most recently complete statewide needs assessment update);
The LIAs that might participate in the PFO initiative; and
The size and demographic characteristics of the populations in at-risk communities in the geographic area that will be included in a PFO initiative.
In your feasibility study, consider:
The evidence-based home visiting model(s) that would be appropriate for implementation as part of a PFO initiative.
The entire landscape of potential providers that can serve the needs of the target population, and their experience in implementing evidence-based home visiting programs.
You should further consider:
The provider’s experience implementing the evidence-based home visiting model;
The provider’s capacity to meet enrollment targets of the PFO initiative, and (if currently a MIECHV-funded LIA), their track record of performance and maintaining enrollment capacity percentage;
The range of referral pathways for recruitment of the target population in order to meet enrollment targets; and
The provider’s capacity to collect and report program data and participate in the PFO evaluation.
The feasibility study should address the potential outcome measure(s) for the PFO initiative, including how they would be measured.31 When determining outcome measure(s) for a MIECHV-funded PFO initiative, ensure that selected measure(s) would meet requirements outlined in Section 3: Outcome Measure(s) and Payments, which include (but are not limited to) required alignment with MIECHV benchmark areas and constructs.
Consider the potential payment amounts for each outcome measure, the payment schedule associated with each, and how it would align with the evaluation reporting timeline.
Ensure payment amounts are reflective of federal, state and/or local cost savings, cost avoidance and/or social benefit, and that they are appropriate and reasonable relative to the outcome measure achieved.
The feasibility study should address your capacity to meet all of the requirements of a rigorous, third-party PFO evaluation as described in Section 4: Third-Party Evaluation of the PFO SIR. In particular, consider:
The capacity and independence of third-party evaluators, as well as your experience engaging with third-party evaluators;
The availability and quality of data to evaluate each outcome measure, including your experience and capacity to access administrative data;
What, if any, data sharing agreements will be needed, and if these agreements already exist;
The recipient’s experience and capacity using data to evaluate, track, and monitor progress on the outcome measure(s) for the PFO initiative; and
Whether the size of the target population is sufficient to be included in the PFO initiative.
In your feasibility study, you should consider:
The anticipated duration of the PFO initiative, including the length of service provision, and the last date that outcome payments are expected to be made;
The amount of time needed to complete the evaluation, determine if outcome payments will be made, and obligate funds; and
If the project, both the intervention and evaluation, can be completed within the 10-year PFO statutory period of availability.
PFO initiatives require the involvement of many partners including LIAs, third-party evaluators, model developers, agencies that house administrative data sources, early childhood systems partners, home visitors, families, and others. They may also include investors and/or an intermediary organization. In the feasibility study, consider:
The potential key stakeholders and partners for a PFO initiative, their level of interest/engagement, and any significant or known barriers to partnership;
The agreements or Memorandums of Understanding (MOUs) that are, or would need, to be in place to implement a PFO initiative; and
The opportunities and challenges associated with engaging home visiting service providers and families in a PFO initiative.
The final step of the feasibility study is to provide an overall assessment as to whether the PFO initiative is or is not determined to be feasible. In making this determination, consider:
The primary benefits and assets associated with implementation of a PFO initiative, as identified through the feasibility study;
The primary risks and challenges associated with implementation of a PFO initiative, as identified through the feasibility study; and
If the PFO initiative is NOT determined to be feasible, consider what steps would be necessary to address the findings should this approach be pursued in the future.
Administrative data: Data collected to help administer a public program.
Administrative expenditures: Refers to the costs of administering a MIECHV grant incurred by the recipient, and includes, but may not be limited to reporting costs, project-specific accounting and financial management, subrecipient monitoring, etc.32
At-risk communities: Communities for which indicators, in comparison to statewide indicators, demonstrated that the community was at greater risk than the state as a whole. At-risk communities are further defined as communities with concentrations of the following indicators: premature birth, low-birth weight infants, and infant mortality, including infant death due to neglect, or other indicators of at-risk prenatal, maternal, newborn, or child health; poverty; crime; domestic violence; high rates of high school dropouts; substance abuse; unemployment; or child maltreatment.33
PFO Project Period: The period of performance for the PFO initiative as proposed by the MIECHV recipient, which may not exceed 10 years.
Cost savings: A reduction in outlay costs. For example, a project yields savings to the federal government if it results in lower federal outlays. This could be the result of dollars not spent because the intervention eliminates a need for the outlay.
Cost avoidance: Cost savings predicted for the future based on action taken.
Recipient-level infrastructure expenditures: Refers to recipient-level expenditures necessary to enable recipients to deliver MIECHV services, but does not include the costs of delivering such home visiting services.
Social Benefit: The total benefit to society of producing a specific outcome, good, or service; the increase in the welfare of a society that is derived from a particular course of action.
Public Burden Statement: HRSA is requesting approval to collect information in response to a Supplemental Information Request (SIR), which will include eligible entities' plans for implementation and evaluation of Pay for Outcomes (PFO) initiatives to be applied for through the MIECHV Program. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this information collection is 0906 -XXXX and it is valid until XX/XX/202X. This information collection is voluntary. Public reporting burden for this collection of information is estimated to average 92 hours per response, including the time for reviewing instructions, searching existing data sources, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to HRSA Reports Clearance Officer, 5600 Fishers Lane, Room 14N136B, Rockville, Maryland, 20857 or [email protected].
1 For example, for any PFO initiatives proposed with FY 2021 MIECHV Formula funds, the PFO SIR Response will be due to HRSA no later than January 28, 2022.
2 If you are proposing to use MIECHV funds only for a PFO feasibility study, you are not required to complete a PFO SIR Response.
3 Social Security Act, Title V, § 511(c).
4 Michalopoulos, C, et. al. (2017). Evidence on the Long-Term Effects of Home Visiting Programs: Laying the Groundwork for Long-Term Follow-Up in the Mother and Infant Home Visiting Program Evaluation (MIHOPE). OPRE Report 2017-73. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
5 Id
6 Id
7 For example, for any PFO initiatives proposed with FY 2021 MIECHV formula funds, the PFO SIR Response will be due to HRSA no later than January 28, 2022.
8 Social Security Act, Title V, § 511(c)(3).
9 Social Security Act, Title V, § 511(c)(4).
10 Social Security Act, Title V, § 511(k)(4).
11 Social Security Act, Title V, § 511(c)(3).
12 Social Security Act, Title V, § 511(k)(4).
13 You are not required to conduct your own cost-benefit analysis. You can reference the relevant literature to satisfy this requirement.
14 Social Security Act, Title V, § 511(k)(4)(B).
15 For more information on the MIECHV benchmark areas and constructs, please refer to MIECHV technical assistance resources.
16 Social Security Act, Title V, § 511(k)(4)(D).
17 Social Security Act, Title V, § 511(j)(3)(B).
18 Social Security Act, Title V, § 511(k)(4).
19 Social Security Act, Title V, § 511(k)(4)(B).
20 Year prior to the initiation of the PFO initiative refers to the fiscal year prior to the FY during which the recipient submits the initial funding application related to the PFO initiative.
21 Social Security Act, Title V, § 511(j)(3)(B).
22The term “administrative expenditures” refers to the costs of administering a MIECHV grant incurred by the recipient, and includes, but may not be limited to reporting costs, project-specific accounting and financial management, subrecipient monitoring, etc.
23 The term “infrastructure expenditures” refers to recipient-level expenditures necessary to enable recipients to deliver MIECHV services, but does not include the costs of delivering such home visiting services.
24 For instructions on how to seek HRSA approval for spending more than 25% of a MIECHV formula award amount on recipient-level infrastructure expenditures, refer to the most recent MIECHV Formula NOFO or NCC Update guidance.
25 Social Security Act, Title V, § 511(k)(4)(D).
26 Key PFO initiative partners include, but are not limited to the third-party evaluator, external funder(s) (if applicable), LIAs/service providers, intermediaries (if applicable), administrative data sources (if applicable), and any other key partners.
27 If you are requesting MIECHV funds for PFO outcome payments and a PFO evaluation as part of a PFO initiative, prior to submitting the PFO SIR Response you must submit as part of your formula funding application one SF-424A budget form and two budget justifications – one for the MIECHV formula award and one for activities associated with a PFO initiative. Each justification must include an explanation of each line item in the budget (e.g., personnel, fringe, travel, equipment, supplies, contractual, other, indirect charges, etc.).
28 Social Security Act, Title V, § 511(c)(3).
29 Year prior to the initiation of the PFO initiative refers to the fiscal year prior to the FY during which the recipient submits the initial funding application related to the PFO initiative.
30 Key PFO initiative partners include, but are not limited to the third-party evaluator, external funder(s) (if applicable), LIAs/service providers, intermediaries (if applicable), administrative data sources (if applicable), and any other key partners.
31 Refer to Section 4: Selected Outcome Measure(s) for detailed requirements.
32 Social Security Act, Title V, § 511(i)(2)(C).
33 Social Security Act, Title V, § 511(b)(1)(A).
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | Draft PFO Guidance_v1 |
Author | Stock, Ann (HRSA) |
File Modified | 0000-00-00 |
File Created | 2021-07-19 |