Overview of the PAF Evaluation

Attachment A

OVERVIEW OF THE PAF study

This page has been left blank for double-sided copying.

OVERVIEW OF THE Pregnancy Assistance Fund (PAF) STUDY

In March 2010, Congress authorized the Pregnancy Assistance Fund Competitive Grants Program as part of the Patient Protection and Affordable Care Act (ACA). The grants program is a key element of the federal strategy to support youth and young adults who are having or raising a child. Administered by the Office of Adolescent Health (OAH), the grants program funded a second cohort of 17 grantees—states, tribes, and tribal entities—in summer 2013 to develop and implement programs focused on an array of outcomes, including increasing access to and completion of secondary and postsecondary education, improving child and maternal health, reducing the likelihood of repeat teen pregnancies, increasing parenting and co-parenting skills, decreasing intimate partner violence, and raising awareness of available resources. To promote positive outcomes, grantees may implement a wide variety of services for expectant and parenting youth, women, fathers, and their families. OAH’s continued investment in programs for expectant and parenting youth has led to their request for a rigorous impact and implementation study of such programs, and they have contracted with Mathematica Policy Research to conduct the Pregnancy Assistance Fund (PAF) Study.

Preliminary PAF Study efforts, including study design and instrument development, will be conducted through a Feasibility and Design Study (FADS). The purpose of the FADS is to design rigorous impact evaluations in three sites that serve pregnant and parenting youth (including Pregnancy Assistance Fund grantees), develop data collection materials for all aspects of an evaluation, and conduct telephone interviews with grantees about the program design decisions and early implementation experiences. The three sites include random assignment sites in California and Texas, as well as a quasi-experimental site in Washington, DC. Information collected through the FADS will also be used to provide funding agencies with information to inform the structure and components of programs for expectant and parenting youth and their families, so that the five-year PAF Study will be possible.

As noted above, the objective of the FADS is to establish a foundation for the PAF Study’s rigorous impact and implementation evaluation. Specifically, FADS will: (1) assess design options for implementation and impact evaluation, (2) document how programs are operationalized in the field, (3) identify and enter into agreements with three sites for the evaluation, (4) provide assistance to sites to support a rigorous evaluation framework, (5) develop all evaluation instruments and obtain clearance, and (6) pilot baseline data collection.

Impact and Implementation Study. Using an experimental design, the PAF Study will test the effectiveness of services to impact educational, health, sexual behavior, and parenting outcomes. During the FADS, the study team will identify and work with two or three sites to decide which services components will be evaluated, which participants will be included, and which outcomes will be measured. In addition, the FADS team will work with sites to develop a plan for random assignment. Finally, the FADS team will work with the selected sites to design a process for collecting study data, including evaluation consent, a baseline survey, and two follow-up surveys.

In each of the two or three sites selected for the impact study, all eligible youth will be considered for enrollment in the study. The evaluation team will then work collaboratively with each site to identify youth for the study and obtain consent.

Active written consent will be obtained from youth older than 18 and parental permission will be obtained for those younger than 18. For youth older than 18, the informed consent process will be integrated with baseline data collection.

The baseline survey will be administered to all consented youth shortly after study enrollment. All consented youth will then be asked to complete two additional follow-up surveys, at 12- and 24-months after the baseline. Youth assent will be collected at the beginning of each survey administration. The mode of data collection will depend on the program setting. We will offer various modes for completing the baseline survey. These modes are likely to be computer-assisted telephone interviewing (CATI) or a self-administered paper and pencil instrument (PAPI). We will work with grantees to assess the best baseline survey mode given the context, with sensitivity to respondent literacy levels and access to technology.

For CATI completes, data collectors will be assigned a project cell phone that will be handed off to respondents during the intake process. Respondents will use the phone to call Mathematica’s Survey Operations Center (SOC) and complete the survey over the phone with a trained interviewer. Data collectors will also be given a toll-free number for the Survey Operations Center that they can give to respondents to call in and complete the survey over the phone at their convenience. Additionally, SOC staff can make calls to the respondent to complete the survey over the phone. When completing the survey through CATI, the interviewer (and data collector, when applicable) will ensure the respondent is in a secure, private place to respond to the survey questions.

A paper and pencil (PAPI) version of the baseline survey will also be available for anyone wishing to complete the survey using the self-administered instrument. Data collectors will keep PAPI versions on-hand and distribute them to respondents, as needed. The completed surveys will be returned to staff in sealed envelopes.

The modes for follow-up survey data collection will include CATI and a web survey as no group administration will be possible given the ongoing sample enrollment of eligible youth in the program sites.

The three programs selected for the impact evaluation will also participate in a more in-depth implementation study. The in-depth implementation study will take a detailed look at program operations along four key aspects: (1) inputs required for implementation to succeed and be sustained, (2) contextual factors that influence implementation, (3) quality of program implementation, and (4) participants’ responsiveness to service.

The in-depth implementation study will collect and analyze data to contextualize the analysis of program impacts. Data will be obtained from the following sources: (1) individual and group discussions with program developers, program leaders and front-line staff, program partners and other stakeholders (Instruments 1 and 2); (2) a paper and pencil survey of frontline staff and supervisors (Instrument 3); (3) group interviews with participating youth (Instrument 4); and (4) a protocol for recording attendance and content coverage (Instrument 5),. Through these data collection efforts, the study will document the program context in each site, the planned intervention, the implementing organization, other organizational partners participating in implementation, implementation systems, youth’s program dosage and youth’s experiences and satisfaction with the programs.

The data will serve two main purposes. First, the information will enable the study team to produce clear, detailed descriptions of each intervention that is evaluated and the counterfactual in each site. This documentation is critical for understanding the meaning of impact estimates. Second, the data will be used to assess fidelity of implementation and the quality of program delivery. This information is essential for determining whether the interventions were implemented well and whether the evaluation provided a good test of each site’s intervention. The in-depth implementation study is the focus of this ICR.

The Study Sites

There are three sites participating in the PAF Study. Two of these sites (California and Texas) will be randomized control trials with primary data collection through surveys of youth. The third site, in Washington, DC, will use a quasi-experimental design and rely on administrative data provided through data use agreements with three local public agencies – DC Public Schools, DC Human Services, and DC Department of Health. Youth will not be surveyed; however, the site will participate in data collection for the in-depth implementation study.

The three sites are described in detail below.

1. California Department of Public Health, Division of Maternal Child and Adolescent Health (MCAH).

CA MCAH is currently an OAH Pregnancy Assistance Fund (PAF) grantee. They are using their PAF grant to introduce the Adolescent Family Life Program Positive Youth Development (AFLP PYD) across their program providers throughout the state. These program providers are currently implementing an older version of the program – AFLP. AFLP PYD differs from the original AFLP in three ways: 1.) Development of structured materials for the case managers to use during interactions with youth, including home visits; 2.) Case managers carry fewer cases and therefore, youth receive double the amount of AFLP dosage via home visits; 3.) Case managers utilize the positive youth development framework, which promotes youth resiliency and self-sufficiency via motivational interviewing and techniques. This study will primarily address the impact of AFLP-PYD, as compared to AFLP, on outcomes such as subsequent pregnancy, improving contraception use, and supporting school completion.

The evaluation will involve approximately 12 current AFLP program providers across the state. Within two of the larger providers, approximately 750 expectant or parenting females will be randomly assigned to either AFLP or AFLP-PYD. Across the remaining 10 providers, we will assign clusters to either AFLP or AFLP-PYD. A cluster may be an entire provider (for example, among the smallest providers), or specific geographic locations served by larger providers. We expect to randomize a total of 14 clusters, and enroll approximately another 800 expectant and parenting females across them. Sample enrollment will occur over an 18-month period.

The evaluation sample is expected to be primarily Hispanic (~80 percent) and low-income (~75 percent of the sample eligible for Medicaid). At enrollment, approximately 55 percent of the sample is expected to be pregnant (and not yet parenting) and 43 percent parenting (and not pregnant). A small percentage (~2 percent) may be pregnant and parenting.

2. The Houston Department of Health and Human Services (HDHHS)

The HDHHS will implement Steps to Success, which was developed by Healthy Families San Angelo (HFSA) as an enhancement to Healthy Families America home visiting services. The primary component of Steps to Success is a structured home visiting model that covers parenting, contraception, employment, relationships, and finances.

HFSA developed Steps to Success based on research on key risk factors for repeat pregnancies among adolescent mothers. This research pointed to the importance of encouraging these young mothers to use long-acting contraceptives (a key element of the Steps to Success approach) as essential to delaying repeat pregnancy. It also suggested that promoting more positive relations with the baby’s father and encouraging these young mothers to stay in school were both promising avenues for reducing the risk of rapid repeat pregnancy. Steps to Success aims to promote both these goals.

Clients are accepted into the program either during the pre-natal period or early post-partum period. Home visits occur weekly initially and transition to monthly visits as appropriate based on the needs of the family. These visits are provided for two years after the baby is born. Home visitors have a maximum case load of twenty-five clients at a time. The study will test the impact of Steps to Success, as compared to a “business as usual” control condition, on outcomes such as delaying a subsequent pregnancy, improving contraception use, supporting school completion, and improving parenting skills.

The evaluation involves randomly assigning eligible and interested young women to Steps to Success or a control group that will have access to any existing community resources (such as healthcare and other related services at WIC clinics) but not to Steps to Success.

We expect to enroll and randomize approximately 575 young mothers over a 24-30 month period. The sample is expected to be primarily Hispanic (~75 percent) and low-income (100 percent having qualified for Medicaid). At enrollment, approximately 50 percent of the sample is expected to be pregnant (and not yet parenting) and 50 percent parenting (and not pregnant).

3. Washington, DC

The Washington, DC-based New Heights program offers school-based individual case management to expectant and parenting youth, plus educational workshops on relevant topics three times a week. Additional components of the program are a youth council and earned incentives for participation. The program is implemented in 13 Washington, DC, public high schools and 2 charter schools. The New Heights program is a former OAH PAF grantee. The quasi-experimental evaluation would focus on the 13 non-charter public high schools.

With this quasi-experimental design we will estimate impacts on educational attainment by simultaneously drawing on two types of comparisons: (1) an eligible-vs.-ineligibles comparison in which we compare youth eligible for New Heights (parenting teens) to youth in the same schools who are not eligible for New Heights (non-parenting youth) and (2) a cohort comparison of youth who attended DC public schools after New Heights was introduced in 2011 to youth who attended DC public schools before New Heights was introduced. The planned analyses will included data from 2007 – 2015. All data for analysis will be acquired through data use agreements with three DC public agencies.