0990-PAF Supporting Statement B_to OMB_082814 revised not highlighted

Part B: Justification for the Collection of Design and Implementation Survey Data and Baseline Data - Pregnancy Assistance Fund (PAF) Feasibility and Design Study (FADS)

April 2014 (Revised August 2014)

Submitted to: U.S. Department of Health and Human Services Office of Adolescent Health Office of the Director Department of Health and Human Services 1101 Wootton Parkway, Suite 700 Rockville, MD 20852 Project Officer: Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Amy Farb

Part B: Justification for the Collection of Design and Implementation Survey Data and Baseline Data - Pregnancy Assistance Fund (PAF) Feasibility and Design Study (FADS) April 2014 (Revised August 2014)

Contents

PART B 1

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for Collection of Information 5

B3. Methods to Maximize Response Rates and Deal with Non-Response 7

B4. Test of Procedures or Methods to be Undertaken 7

TABLES

Table B1.1. Minimum Detectible Impacts for California 3

ATTACHMENTS

ATTACHMENT A: OVERVIEW OF THE PAF EVALUATION

ATTACHMENT B: EMAIL TEMPLATE FOR INTERVIEW REQUEST

ATTACHMENT C: GRANTEE INTERVIEW GUIDE

ATTACHMENT D: PAF BASELINE SURVEY

ATTACHMENT E: QUESTION BY QUESTION SOURCE LIST FOR THE BASELINE SURVEY

ATTACHMENT F: SOURCES REFERENCED FOR THE BASELINE SURVEY

ATTACHMENT G: BASELINE PRETEST MEMO

ATTACHMENT H: 60 DAY FEDERAL REGISTER NOTICE

ATTACHMENT I: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT AND/OR ANALYSIS OF THE PAF BASELINE SURVEY

ATTACHMENT J: CONSENT LETTERS AND FORMS AND YOUTH ASSENT FORM

ATTACHMENT K: CONFIDENTIALITY PLEDGE

ATTACHMENT L: ANALYSIS PLAN

ATTACHMENT M: OVERVIEW DOCUMENT FOR GRANTEES

ATTACHMENT N: 30 DAY FEDERAL REGISTER NOTICE

PART B

In March 2010, Congress authorized the Pregnancy Assistance Fund (PAF) Competitive Grants Program as part of the Patient Protection and Affordable Care Act (ACA). The PAF grant program is a key element of the federal strategy to support teens and young adults who are having or raising a child and their families. Administered by the Office of Adolescent Health (OAH), the PAF grant program funded a second cohort of 17 grantees—states, tribes, and tribal entities—in summer 2013 to develop and implement programs focused on an array of outcomes, including increasing access to and completion of secondary and postsecondary education, improving child and maternal health, reducing the likelihood of repeat teen pregnancies, increasing parenting and co-parenting skills, decreasing intimate partner violence, and raising awareness of available resources. To promote positive outcomes, grantees may implement a wide variety of services for expectant and parenting teens, women, fathers, and their families.

The PAF evaluation will have two core components: a rigorous assessment of program impacts and implementation of two or three grantees, and a descriptive examination of program design of all 17 grantees. The PAF evaluation will help the federal government, grantees, and local service providers learn more about program design, implementation, and impacts.

Preliminary PAF evaluation efforts, including instrument development, will be conducted through the PAF Feasibility and Design Study (FADS). The purpose of the FADS is to design rigorous impact evaluations in two sites, develop data collection materials for all aspects of a future evaluation, and conduct telephone interviews with grantees about the program design decisions and early implementation experiences. Information collected through the FADS will also be used to provide funding agencies with information to inform the structure and components of programs for expectant and parenting teens and their families, so that future rigorous program evaluation will also be possible.

The FADS comprises a design and implementation analysis and an impact study as its two primary data collection components. The design and implementation analysis will describe grantees’ program design and factors that influenced their decision making. The impact study will use an experimental design to test the effectiveness of PAF-funded services on outcomes related to education, sexual behaviors, parenting, and health. An overview of the PAF Evaluation is found in Attachment A.

This proposed information collection activity focuses on collecting (a) program design and early implementation data collected through telephone interviews with grantees and (b) baseline data collection in up to three impact sites.

B1. Respondent Universe and Sampling Methods

Design and Implementation Analysis. In summer 2013, 17 states and tribal entities received Pregnancy Assistance Funds (PAF) grants. We expect one respondent from each state or tribal entity to be contacted and interviewed once over the course of two years. Respondents will be grantee-level PAF administrators. The specific individuals to be interviewed will be identified through a review of the grantee applications, and confirmed by federal PAF grantee project officers.

Impact Study. OAH will select two program sites to participate in an experimental evaluation, at least one of which is a current PAF grantee. The sites are not meant to be representative of PAF-funded programs as a whole. Site selection has focused on grantees that (1) are large enough to support an impact study, (2) are implementing programs appropriate for PAF in a way that is amenable to random assignment, and (3) address priority gaps in the existing research literature on evidence-based approaches to assist pregnant and parenting youth.

The selected sites, pending final OAH approval, are (1) California Department of Health, Division of Maternal, Child, and Adolescent Health (MCAH) and (2) The Texas Children’s Health Plan (TCHP). At this time, we do not anticipate that there will be a third random assignment site.

MCAH

CA MCAH is currently an OAH Pregnancy Assistance Fund (PAF) grantee. They are using their PAF grant to introduce the Adolescent Family Life Program Positive Youth Development (AFLP PYD) across their program providers throughout the state. These program providers are currently implementing an older version of the program – AFLP. AFLP PYD differs from the original AFLP in three ways: 1.) Development of structured materials for the case managers to use during interactions with youth, including home visits; 2.) Case managers carry fewer cases and therefore, youth receive double the amount of AFLP dosage via home visits; 3.) Case managers utilize the positive youth development framework, which promotes youth resiliency and self-sufficiency via motivational interviewing and techniques.

This study will address the following primary question:

• Compared to the existing AFLP, is the AFLP PYD program more successful at delaying a subsequent pregnancy, improving contraception use, and supporting school completion?

The evaluation will involve 12 current AFLP program providers across the state. Within two of the larger providers, approximately 750 expectant or parenting females will be randomly assigned to either AFLP or AFLP-PYD. Across the remaining 10 providers, we will assign clusters to either AFLP or AFLP-PYD. A cluster may be an entire provider (for example, among the smallest providers), or specific geographic locations served by larger providers. We expect to randomize a total of 14 clusters, and enroll approximately another 800 expectant and parenting females across them. Sample enrollment will occur over an 18-month period.

The evaluation sample is expected to be primarily Hispanic (~80 percent) and low-income (75 percent of the sample eligible for Medicaid). At enrollment, approximately 55 percent of the sample is expected to be pregnant (and not yet parenting) and 43 percent parenting (and not pregnant). A small percentage (~2 percent) may be pregnant and parenting.

Youth will be surveyed three times – at the time of study enrollment (baseline), 12-months later, and 24-months later.¹ The primary mode of survey completion for all survey rounds is Computer Assisted Telephone Interviewing (CATI); 80 percent of all completes at all rounds are expected to complete the survey using CATI. Youth will also be given the option to complete the survey on a hard copy (PAPI) or online.

An overall impact will be calculated as a weighted average of the impacts from the two designs. We will use inverse variance weights in our benchmark analysis and sample size weights as a sensitivity analysis. At the time of the 24-month follow-up, we expect to retain 75 percent of the sample. For a prevalence rate of 25 percent (such as a subsequent pregnancy during the follow-up period), we can detect a 7 percentage point difference between the two groups; and, for a prevalence rate of 50 percent (such as receiving a diploma during the follow-up period), we can detect a 9 percentage point difference between the two groups. Examining impacts by particular sub-groups (such as whether expecting or parenting at program enrollment, or whether primary language is English or Spanish) will be considered exploratory, as the study is not considered sufficiently powered to detect impacts on those samples. Given the risk profile of the population, the findings from this study will have policy relevance for the field without sub-group analysis.

Table B1.1 reports minimum detectible impacts on two illustrative outcomes—one with 50 percent prevalence and one with 25 percent prevalence. Separate estimates are presented for the two components of the evaluation (individual randomization and cluster randomization) as well as for the overall study (in which the overall impact is calculated as a weighted average of the impacts from the two study components).

Table B1.1. Minimum Detectible Impacts for California

	Percentage Point Impacts for Illustrative Binary Outcomes
Study Component	50 percent prevalence rate	25 percent prevalence rate
Individual Randomization (2 sites; 550 youth)	10.6	8.6
Cluster Randomization (14 sites; 600 youth)	18.2	14.3
Full Study	8.9	7.1

Notes:   Sample sizes account for survey nonresponse. Figures assume that the sample is evenly divided between the program and control groups and that covariates explain 20 percent of the variance at the individual level and 40 percent at the cluster level. We assume an ICC of 0.06. The figures also assume a two-tailed t-test with 80 percent power and a 95 percent confidence interval. 

TCHP²

TCHP currently operates a home visiting program for expectant and parenting young women who qualify for Medicaid. For the evaluation, they will replace that program with a new model that focuses on delaying a subsequent pregnancy and involves the fathers in home visits. This new model, Steps to Success, was developed by Healthy Families San Angelo (HFSA) as an enhancement to Healthy Families-style home visiting services. The primary component of Steps to Success is a structured home visiting model that covers parenting, contraception, employment, relationships, and finances.

HFSA developed Steps to Success based on research on key risk factors for repeat pregnancies among adolescent mothers. This research pointed to the importance of encouraging these young mothers to use long-acting contraceptives (a key element of the Steps to Success approach) as essential to delaying repeat pregnancy. It also suggested that promoting more positive relations with the baby’s father and encouraging these young mothers to stay in school were both promising avenues for reducing the risk of rapid repeat pregnancy. Steps to Success aims to promote both these goals.

Clients are accepted into the program either during the pre-natal period or early post-partum period. Home visits occur weekly initially and transition to monthly visits as appropriate based on the needs of the family. These visits are provided for two years after the baby is born. Home visitors have a maximum case load of twenty-five clients at a time.

This study will address the following primary question:

• Compared to a control condition, is Steps to Success more successful at delaying a subsequent pregnancy, improving contraception use, supporting school completion, and improving parenting skills?

The evaluation involves randomly assigning eligible and interested young women to Steps to Success or a control group that will have access to any existing community resources but not to Steps to Success. The control group youth will also receive small gifts (such as diapers, formula, and gift cards) four times, within the first year after random assignment. These gifts are valued at $30 per quarter or $120 per control group member per year. Site staff believe that giving some minimal benefit, unrelated to the treatment program, to the control group is necessary to keep control group members committed to the study through follow up.

We expect to enroll and randomize approximately 575 young mothers over a 24-30 month period. The sample is expected to be primarily Hispanic (~75 percent) and low-income (100 percent having qualified for Medicaid). At enrollment, approximately 75 percent of the sample is expected to be pregnant (and not yet parenting) and 25 percent parenting (and not pregnant).

As part of a follow-up contract, youth will be surveyed three times – at the time of study enrollment (baseline), 12-months later, and 24-months later. The primary mode of survey completion for all survey rounds is Computer Assisted Telephone Interviewing (CATI); 80 percent of all completes at all rounds are expected to complete the survey using CATI. Youth will also be given the option to complete the survey on a hard copy (PAPI) or online.

At the time of the 24-month follow-up, we expect to retain 75 percent of the sample. For a prevalence rate of 25 percent (such as a subsequent pregnancy during the follow-up period), we can detect a 10 percentage point difference between the two groups; and, for a prevalence rate of 50 percent (such as receiving a diploma during the follow-up period), we can detect a 12 percentage point difference between the two groups. Given the small sample size, we do not anticipate conducting any subgroup analyses.

We anticipate sufficient power to detect policy relevant differences. About 60 percent of at-risk teens report using a highly effective contraceptive method.³ We will be able to detect about a 12 percentage point improvement in effective contraceptive use. In the Building Strong Families study we found that about 60 percent of mothers reported that the father contributed to the child financially. We would likewise be able to detect about a 12 percentage point improvement in that parenting measure. These calculations assume binary outcomes, no clustering, a two-tailed hypothesis test and no perceived benefit from covariate adjustment. Not being able to provide subgroup comparisons (by race or ethnicity, for example) would still allow this data collection to inform policy around services to expectant and parenting youth.

B2. Procedures for Collection of Information

Design and Implementation Analysis. The data collection modality will be telephone interviews.

OAH has provided Mathematica Policy Research with a key contact for each grantee receiving PAF funding. The contractor will send an email to the key contact that describes the Design and Implementation Analysis, and the purpose of and nature of the questions on the grantee interview protocol for which an interview is being requested. Attached to that email will be a document that provides an overview of the entire PAF evaluation, including the Design and Implementation component (see Attachment M). The email will ask the intended respondent to select a day and time for an interview that will not last longer than two hours. The email will also suggest that the key contact recommend a respondent, if the key contact does not believe they are in the best position to address the grantee interview protocol questions. Telephone follow-up will be initiated if the key contact does not respond within three business days.

The specific questions asked during each interview will vary, depending on 1) what is already known about the respondent’s program design decisions (e.g. we will not ask questions for which we already have answers, based on Mathematica’s prior review of program documents and administrative data) and 2) the discretion of the interviewer, who will adapt his or her questions based on the respondent’s answers, while still touching on key themes across interviews.

A specific protocol will be developed for each interview in advance of the call. This overall approach has been used for the PREP evaluation, and has demonstrated benefit of reducing burden for respondents, because each interview will be tailored for the specific respondent who is being interviewed. Attachment B includes a copy of the email template for requesting an interview. A copy of the grantee interview protocol is found in Attachment C.  

Contractor staff will take notes during discussions, obtain relevant written materials that are readily available, and prepare written summaries of each interview.

Impact Study. In each of the two sites selected for the impact study, all eligible youth will be considered for enrollment in the study (discussed in Section B.1). Each site will be responsible for providing the evaluation team with a list of eligible youth. The evaluation team will then work collaboratively with each site to identify youth for the study and obtain consent.

Mathematica will thoroughly and efficiently train site staff to ensure they can properly inform study participants. We will create a brief study description to ensure that accurate and consistent information is available, and train program staff on explaining the study, reviewing the study description, answering questions about the study, and administering consent (if applicable for the site intake process) and the baseline survey (if applicable for the site intake process). We will provide instruction on these generic aspects of the study procedures through in-person meetings and webinars, depending on the content of each training.

Active written consent will be obtained from youth older than 18 and parental permission will be obtained for those younger than 18. For youth older than 18, the informed consent process will be integrated with baseline data collection. For younger youth, the evaluation team and program staff will collaborate on a process for sharing the consent forms and a study description with parents and guardians that is integrated with their program intake process. For example, youth may take study forms home to share with their parents/guardians. Parents/guardians would then have the option of consenting by paper or by calling the toll-free study help line. Alternatively, study settings could host periodic informational meetings for interested youth and their parents, at which time parental permission could be requested and obtained. Program staff will return all consent forms to Mathematica through FedEx. Attachment J provides an example of the consent and assent forms.

The baseline survey will be administered to all consented youth shortly after study enrollment. Youth assent will be collected at the beginning of the baseline administration from those under 18. We will offer various modes for completing the baseline survey. These modes are likely to be computer-assisted telephone interviewing (CATI), a web-survey, or a self-administered paper and pencil instrument (PAPI). We will work with grantees to assess the best baseline survey mode given the context, with sensitivity to respondent literacy levels and access to technology.

For CATI completes, data collectors will be assigned a project cell phone that will be handed off to respondents during the intake process. Respondents will use the phone to call Mathematica’s Survey Operations Center (SOC) and complete the survey over the phone with a trained interviewer. Data collectors will also be given a toll-free number for the Survey Operations Center that they can give to respondents to call in and complete the survey over the phone at their convenience. Additionally, SOC staff can make calls to the respondent to complete the survey over the phone. When completing the survey through CATI, the interviewer (and data collector, when applicable) will ensure the respondent is in a secure, private place to respond to the survey questions. We have used this method successfully in the past for similar evaluations that similarly ask sensitive questions. CATI has the added benefit of ensuring higher rates of survey completion, with a “live person” walking through the survey with respondents.

We will also offer the option of completing the survey over the web. If a web-based survey is used, respondents will be provided a unique PIN/password. We will also provide them with a toll-free number to call should they have any issues with the web survey.

A paper and pencil (PAPI) version of the baseline survey will also be available for anyone wishing to complete the survey using the self-administered instrument. Data collectors will keep PAPI versions on-hand and distribute them to respondents, as needed. The completed surveys will be returned to staff in sealed envelopes

Attachment D provides a copy of the baseline survey. A question by question list of sources for the baseline survey is found in Attachment E, and a description of the sources referenced is found in Attachment F.

B3. Methods to Maximize Response Rates and Deal with Non-Response

Design and Implementation Analysis. We expect to interview one grant administrator for each of the 17 PAF grantees, achieving a 100 percent response rate among grantees. Several factors will help ensure a high rate of cooperation among respondents. First, grantees are required to participate in evaluation activities per the requirements of their grant award.⁴ Individuals may opt out of these interviews; however, grantees that received PAF funds are expected to make someone available who can complete the interview. Second, the stakeholders who will be interviewed or surveyed are all heavily invested in the issues surrounding teen pregnancy assistance and thus should be motivated to participate in the interview. Third, federal staff (PAF project officers) will contact grantees who do not respond to interview requests and ask them to make an official available who can participate.

Impact Study. OAH expects to achieve a response rate of 95 percent for the baseline survey. A high response rate is expected because survey administration will occur shortly after consent is received, and at the time that youth are voluntarily seeking out pregnancy and parenting assistance services. This timing will ensure our contact data are current (no location problems) and that surveys can be administered to most youth in the location where the program will take place.

In addition, we expect that obtaining the sites’ willing assistance will be very important to maximizing the response rate; we will therefore invest significant effort in gaining their cooperation, minimizing burden on sites, integrating an effective consent process, and assuring privacy to the youth participants. Sites will be given detailed information about the surveys, how they will be administered and on what schedule, and how data will be used and protected. Bringing sites into the process while minimizing burden on them will assure site support of the PAF baseline data collection.

Methods to achieve high response rates at follow-up will be discussed in future information collection requests.

B4. Test of Procedures or Methods to be Undertaken

Design and Implementation Analysis. The information collection instruments are similar to discussion protocols that have been used successfully in prior studies, such as the PREP evaluation.

Impact Study. As discussed in Part A of this information collection request, many of the items included on the baseline survey are taken directly from similar surveys OMB has already approved for use in the ongoing Evaluation of Adolescent Pregnancy Prevention Approaches (PPA), the Teen Pregnancy Prevention Replication Study and the Personal Responsibility Education Program (PREP) Multi-Component Evaluation⁵. To date, the PPA baseline survey has been administered to approximately 6,270 adolescents, including 1,535 expectant and parenting young women; the Replication Study baseline has been administered to 7,945 adolescents; and the PREP baseline has been administered to 1,414 youth, including 148 expectant and parenting young women. HHS has made a priority of aligning measures being used in other federal evaluation will also be conducted by Mathematica Policy Research, the contractor for this study, on behalf of OAH.

The baseline survey was pretested with a sample of nine youth participating in a program for parenting teens in Chicago, IL. The pretest confirmed our burden estimates, and resulted in minor wording changes, but no substantive changes to the (1) the length of time needed to complete the questionnaire, (2) our instructions, and (3) the skip logic. The pretest respondents had little trouble completing the instruments and following directions as instructed. Attachment G includes a copy of the pretest memo, which details the pretest procedures and summarizes adjustments made to the baseline survey as a result of the pretest.

1 The current ICR only pertains to the baseline survey.

2 Final approval for the TCHP site is pending senior management approval at the site. We will alert OMB immediately if the site declines participation.

3 Jones J, Mosher WD and Daniels K, Current contraceptive use in the United States, 2006–2010, and changes in patterns of use since 1995, National Health Statistics Reports, 2012, No. 60, <http://www.cdc.gov/nchs/data/nhsr/nhsr060.pdf>, accessed Mar. 20, 2013.

4 See the State Personal Responsibility Education Program (PREP) Funding Opportunity Announcement (p. 11), available at http://www.acf.hhs.gov/grants/open/foa/view/HHS-2010-ACF-ACYF-PREP-0125.

5 ACF received initial OMB approval for the PPA baseline survey on July 26, 2010 (OMB Control Number 0970-0360). In summer 2011, oversight of PPA was transferred to the Office of Adolescent Health (OAH) within the Office of the Assistant Secretary, and the project is now tracked with a different OMB Control Number (0990-0382). The OMB Control Number for the Teen Pregnancy Prevention Replication Study is 0990-0394. OMB approval for the PREP baseline survey was received on March 12, 2013 (OMB Control Number 0970-0398).