Attachment M Overview For Grantees

ATTACHMENT M
OVERVIEW FOR GRANTEES

P A Pregnancy
F Assistance
Fund

Pregnancy Assistance Fund (PAF)
Feasibility and Design Study

Feasibility and Design Study

The Pregnancy Assistance Fund (PAF) Competitive Grants
Program, established as part of the Patient Protection and
Affordable Care Act, is a key element of the federal strategy
to support teens and young adults who are having or raising
a child. Administered by the U.S. Department of Health
and Human Service’s Office of Adolescent Health (OAH),
the PAF program funded a second cohort of 17 grantees—
states, tribes, and tribal entities—in the summer of 2013
to develop and implement programs focused on an array of
outcomes, including access to and completion of secondary
and postsecondary education, child and maternal health,
likelihood of repeat teen pregnancies, parenting and coparenting skills, and intimate partner violence. To promote
positive outcomes, grantees may implement a wide variety
of services for expectant and parenting teens, women,
fathers, and their families.
The PAF evaluation will help the federal government,
grantees, and local service providers learn more about
program design, implementation, and outcomes. All PAF
grantees will be part of an implementation study by sharing their program design decisions and implementation
experiences. OAH also expects to engage two or three
grantees in a rigorous impact and in-depth implementation evaluation. OAH has contracted with Mathematica
Policy Research to lead preliminary study efforts through
the Feasibility and Design Study (FADS).
As a first step, the FADS team will hold discussions with
grantees about their programs to determine whether
conditions are conducive for the impact evaluation. In
particular, the project team will explore whether the most
rigorous design possible—involving random assignment
of youth, groups of youth, or sites—might align with or
only minimally alter existing program implementation
plans. This is most often possible when more individuals
or groups are interested in or eligible for the program
than can be served at any one time. Random assignment
does not mean denial of key program services to individuals in need. The experienced FADS team will be able to

December 2013
The PAF Impact Evaluation:
A Unique Opportunity for Selected Sites
Participating in the impact evaluation will offer three key
benefits to grantees:
1. A clear answer to the question “How much of a
difference is my program making?” The impact study
will use the most rigorous research methods possible in
a given program and will offer evidence of the effects
the program is having on the lives of participating youth
and their families. It will answer the following questions:
How successful was the program overall in changing key
outcomes? Is the program more successful with certain
groups of youth than with others? Is it more likely to
change certain kinds of outcomes than others?
2. A fair and constructive way to allocate PAF program
services. For many program providers, available resources
will limit services to a smaller group of expectant and
parenting teens and young adults than the total number
who may be eligible for or interested in the program. The
study will use a carefully implemented random program
allocation process to give all eligible or interested youth
an equal chance to participate in the program, instead of a
more traditional first-come, first-served approach. In some
cases, those not randomly selected for the program initially
could be scheduled to receive services later, after evaluation
data collection ends, as individual program spaces open up
or in line with a grantee’s plans to expand the program and
roll it out to new sites in the later years of the grant.

3. An opportunity to understand implementation from
the perspective of a supportive and friendly outside
observer. Grantees selected for the impact evaluation
will have their implementation studied more closely than
will the other grantees. This in-depth study of program
activities can help grantees by providing an objective and
constructive examination of processes, challenges, and
solutions. This evaluation component will not grade success,
but it will offer lessons learned to both participating grantees
and the broader service community regarding pregnancy
assistance and parenting programs.

recognize conditions and opportunities that will support
this kind of rigorous evaluation, without significantly
disrupting program implementation plans.
Based on these discussions with grantees, OAH will
select sites for the impact evaluation. The FADS team
will then work with leadership at the selected sites to
develop a plan for the impact evaluation.

Key Elements of the PAF Evaluation:
How They Work
The PAF evaluation will have two core components: a
rigorous assessment of program impacts and a descriptive
examination of program implementation.
Measuring Program Impacts
During the FADS, Mathematica’s team will identify and
work with two or three grantees to decide which services
components will be evaluated, which participants will
be included, and which outcomes will be measured. No
single research question will necessarily guide the impact
study for all the participating grantees. For example, for
one grantee the study may focus on services for young
mothers and measure their college and career readiness,
while for another it may focus on services for young
fathers and measure their parenting skills and attachment
to their children.
In addition, the FADS team will work with grantees
to develop a plan for random assignment. If far more
individuals are eligible for services than the program
can accommodate at one time, individuals randomly
selected for the treatment group may be able to receive
certain services now, while individuals randomly selected
for the comparison group may be able to receive those
services later, or receive different services. If the grantee
plans to increase services over time at additional
locations, such as districts or individual schools, eligible
districts or schools could be randomly selected to
implement services sooner versus later. These are just
two examples of how the study team can design random assignment to work in concert with grantees’ plans
and preferences. The study team will also help grantees
to explain the evaluation to local stakeholders.

Finally, the FADS team will work with the selected sites
to design a process for collecting study data, including evaluation consent, intake surveys, and long-term
follow-up surveys. The FADS team will work collaboratively with grantee officials to ensure that the surveys
capture the program’s desired outcomes and the consent
processes align with program intake procedures. The
evaluation team, not the grantees, will assume most of
the burden for data collection activities.
Documenting Program Implementation
All PAF grantees will participate in one component of the
implementation study—telephone interviews about program
design and implementation experiences. These interviews
will be conducted toward the end of the first grant year. The
two or three grantees selected for the impact evaluation will
participate in a more in-depth implementation study. This
component will take a detailed look at program operations.
The study team will focus on four key aspects of implementation: (1) inputs required for implementation to succeed
and be sustained, (2) contextual factors that influence implementation, (3) quality of program implementation, and (4)
participants’ responsiveness to service. Data collection methods will include phone interviews with program officials;
site visits that may involve interviews with staff, focus groups
with participants, and direct observation of service delivery;
and monitoring of administrative information such as program enrollment, attendance, cost, and service intensity.

For More Information
Mathematica Project Leadership
Matthew Stagner, Project Director
[email protected]
Susan Zief, Deputy Project Director
[email protected]

Federal Project Leadership
Amy Feldman Farb, Evaluation Specialist
Office of Adolescent Health,
Department of Health and Human Services
[email protected]

The PAF Feasibility and Design Study (FADS) is being conducted by Mathematica Policy Research
under contract #HHSP233201300426G.