Attachment D - PREP Evaluation Description - 2-1-12

PREP

Personal Responsibility Education
Program (PREP) Evaluation
December 2011

The Personal Responsibility Education Program (PREP),
funded under the Affordable Care Act of 2010, is a key
element of a multi-pronged federal strategy to reduce
teenage pregnancies and sexually transmitted infections
(STIs). PREP provides grants to states to replicate, or
substantially incorporate elements of, programs that have
been shown to be effective at delaying sexual initiation,
reducing pregnancy, and increasing contraceptive use
among sexually active youth. The selected programs
must educate youth on both abstinence and contraception.
Another aspect of the PREP program is that states must
incorporate lessons on at least three of the following
adulthood preparation subjects: adolescent development,
healthy relationships, healthy life skills, parent-child
communications, educational and career success, and
financial literacy. Forty-five states, Micronesia, Puerto
Rico, the Virgin Islands, and the District of Columbia
have been awarded PREP state grants. Programs will
be available for up to five years to youth ages 10 to 19,
beginning in 2011.
Upon authorizing the PREP program, Congress required
that it be evaluated. The evaluation will help the federal
government, states, and local service providers learn
more about program design, implementation, and outcomes. The Administration for Children and Families
(ACF) within the U.S. Department of Health and Human
Services (DHHS) has contracted with Mathematica
Policy Research to document and assess this large-scale
replication effort. The evaluation will study PREP programs
nationwide, collecting data on programs operating in all
states, and will also conduct an in-depth examination
of a few selected PREP sites.

Across all states, the PREP evaluation will:
• Document how state PREP programs are
designed and implemented. States must
decide how to distribute grant funds to local
sub-awardees, what program models to authorize, and how to support implementation.
Understanding the range of programming decisions made and how implementation unfolds
can contribute to stronger future distribution of
federal grant funds for pregnancy prevention
and other programs.
• Measure and report on program performance.
All state PREP grantees will be required to
submit standard information on program
delivery, participant characteristics, and
fidelity to program models. These measures
will be used to monitor program performance
and identify areas for program improvement.
Within a small number of selected sites,
the PREP evaluation will:
• Examine the impacts and implementation
of PREP. In four or five sites, the evaluation
will provide rigorous estimates of program
effectiveness on key outcomes, such as rates of
sexual initiation and abstinence, contraceptive
use, and teen pregnancy. The evaluation will
also include a detailed look at program delivery.

PREP Nationwide
Design and Implementation Study. Documentation of states’
PREP design decisions and implementation experiences will be
based on two rounds of data collection. In early 2012, the evaluation team will begin a Design Survey. The team will review
documents that outline state PREP program plans and then conduct telephone interviews with state PREP officials to learn more
about what programming decisions have been made, and why. In
mid-2013, the team will begin an Implementation Survey, which
will focus on program implementation, including training and
technical assistance provided, monitoring activities to ensure that
programs are being replicated with fidelity, adaptations made to
fit local contexts, and youth enrollment and retention.
Performance Analysis Study. Performance measures, collected
consistently from all state PREP grantees beginning in fall 2012,
will help ACF understand whether the PREP objectives are being
met and whether technical assistance may be needed to support
program improvement. The PREP evaluation team will:
• Develop performance measures that are aligned with items
used to assess other federally funded teenage pregnancy
prevention programs. The team will also modify the existing
measures to reflect the unique features of PREP.
• Disseminate the processes for collecting data from participants and programs, and explain how states will submit
data through an online data collection system.
• Provide training and technical assistance to states and
their sub-awardees on how to fulfill performance reporting
requirements, beginning in spring 2012.

study, the evaluation team will work collaboratively with grantees
to develop a plan for either randomly assigning individuals to a
group that will receive the program or to a control group, or randomly assigning program sites (such as schools, clinics, or group
homes) to deliver the program or to be control sites. Under both
scenarios, youth in the control group will be able to participate in
the program after data collection for the evaluation is complete.
The evaluation team will administer surveys to study youth shortly
before the programs begin, and about 6 and 18 months after the
programs end. Using data from these two follow-up surveys,
the evaluation team will compare outcomes for youth randomly
assigned to the programs and those who were not; differences
detected between these two groups can be attributed to the PREP
programs. The evaluation design and planned analyses will provide sound, scientific evidence about program effectiveness within
the context of this large-scale replication effort.
The implementation study will focus a scientific lens on
program operations to identify factors that affect large-scale
replication of program models. The evaluation team will
make multiple visits to the selected sites, conduct interviews
with stakeholders and program staff, hold focus groups with
program participants, and review program documents. Team
members will also observe program sessions to document the
quality of delivery and fidelity to the program models.

For More Information
Mathematica Project Leadership
Robert Wood (609-936-2776)
[email protected]

PREP In-Depth

Susan Zief (609-275-2291)
[email protected]

Impact and Implementation Study. In four or five sites
(a site could be an entire state or local sub-awardees within
a state), the evaluation will take a detailed look at how PREP
programs are implemented and whether they are effective.

Mathematica Policy Research
600 Alexander Park
Princeton, NJ 08543
www.mathematica-mpr.com

The impact study will primarily aim to determine whether
the selected programs are effective at reducing teen pregnancy,
STIs, and sexual risk behaviors. It will also seek answers to
the following questions: Were the programs more effective for
particular subgroups of participants? What intermediate changes
in youth attitudes, beliefs, and behaviors may be related to
impacts on pregnancy, STIs, and sexual risk behaviors? Do
impacts vary by particular features of program implementation
or attendance?

Federal Project Leadership

The impact study will be based on a random assignment design
(an experimental evaluation). For sites that can support random
assignment and are selected for the impact and implementation

Clare DiSalvo (202-401-4537)
[email protected]
Administration for Children and Families
Office of Planning, Research, and Evaluation
901 D Street SW, 7th Floor
Washington, DC 20447
Dirk Butler (202-205-5754)
[email protected]
Administration for Children and Families
Family and Youth Services Bureau
1250 Maryland Avenue, SW, Rm 8313
Washington, DC 20024

The PREP Evaluation is being conducted by Mathematica Policy Research and
its subcontractors—Child Trends, Twin Peaks Partners, Decision Information
Resources, Public Strategies, and Chapin Hall—under contract with ACF, DHHS.