PAF Supporting Statement B_InDepth Study_Revised120216_track changes

Part B: Justification for the Collection of In-depth Implementation Study Data - Positive Adolescent Futures (PAF) Study (0990-0428)

February 2015

Revised December 2016

-

Submitted to: U.S. Department of Health and Human Services Office of Adolescent Health Office of the Director Department of Health and Human Services 1101 Wootton Parkway, Suite 700 Rockville, MD 20852 Project Officer: Amy Farb Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Amy FarbSusan Zief

Part B: Justification for the Collection of In-depth Implementation Study Data - Positive Adolescent Futures (PAF) Study February 2015 Revised December 2016

Contents

PART B Introduction 1

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for Collection of Information 5

B3. Methods to Maximize Response Rates and Deal with Non-Response 7

B4. Test of Procedures or Methods to be Undertaken 7

ATTACHMENTS

ATTACHMENT A: OVERVIEW OF THE PAF STUDY

ATTACHMENT B: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT

ATTACHMENT C: CONSENT LETTERS AND FORMS AND YOUTH ASSENT

FORM FOR FOCUS GROUPS

ATTACHMENT D: CONFIDENTIALITY PLEDGE

ATTACHMENT E: QUESTION by QUESTION SOURCES FOR THE STAFF SURVEY

ATTACHMENT F: 60-DAY FEDERAL REGISTER NOTICE

INSTRUMENTS

INSTRUMENT #1 : MASTER LIST OF TOPICS FOR IMPLEMENTATION STUDY

INSTRUMENT #2 : MASTER TOPIC GUIDE FOR FOR SMALL GROUP INTERVIEWS

WITH STAFF

INSTRUMENT #3 : STAFF SURVEY

INSTRUMENT #4 : TOPIC GUIDE FOR FOCUS GROUP DISCUSSION WITH

PARTICIPATING YOUTH

INSTRUMENT # 5: PROTOCOL TO COLLECT ATTENDANCE AND CONTENT

COVERAGE DATA

PART B INTRODUCTION

In March 2010, Congress authorized the Pregnancy Assistance Fund Competitive Grants Program as part of the Patient Protection and Affordable Care Act (ACA). The grants program is a key element of the federal strategy to support youth and young adults who are having or raising a child. Administered by the Office of Adolescent Health (OAH), the grants program funded a second cohort of 17 grantees—states, tribes, and tribal entities—in summer 2013 to develop and implement programs focused on an array of outcomes, including increasing access to and completion of secondary and postsecondary education, improving child and maternal health, reducing the likelihood of repeat teen pregnancies, increasing parenting and co-parenting skills, decreasing intimate partner violence, and raising awareness of available resources. To promote positive outcomes, grantees may implement a wide variety of services for expectant and parenting youth, women, fathers, and their families. OAH’s continued investment in programs for expectant and parenting youth has led to their request for a rigorous impact and implementation study of such programs, and they have contracted with Mathematica Policy Research to conduct the Positive Adolescent Futures (PAF) Study.

Preliminary PAF Study efforts, including study design and instrument development, will be conducted through a Feasibility and Design Study (FADS). The purpose of the FADS is to design rigorous impact evaluations in three sites that serve pregnant and parenting youth (including Pregnancy Assistance Fund grantees), develop data collection materials for all aspects of an evaluation, and conduct telephone interviews with grantees about the program design decisions and early implementation experiences. Information collected through the FADS will also be used to provide funding agencies with information to inform the structure and components of programs for expectant and parenting youth and their families, so that the five-year PAF Study will be possible.

The objective of the Feasibility and Design Study (FADS) is to establish a foundation for the Positive Adolescent Futures (PAF) Study rigorous impact and implementation evaluation. Specifically, FADS will: (1) assess design options for implementation and impact evaluation, (2) document how programs are operationalized in the field, (3) identify and enter into agreements with three sites for the evaluation, (4) provide assistance to sites to support a rigorous evaluation framework, (5) develop all evaluation instruments and obtain clearance, and (6) pilot baseline data collection. Attachment A provides an overview of the components of the PAF Study, which the FADS work is supporting.

Current Information Clearance Request. With this new ICR, OAH is requesting OMB approval for instruments related to the PAF In-Depth Implementation Study.

1. The Master List of Topics for Staff Interviews (Instrument 1)

2. The Topic Guide for Group Discussions with Front-line Staff (Instrument 2)

3. The Staff Survey (Instrument 3)

4. The Topic Guide for Focus Group Discussions with Participating Youth (Instrument 4)

5. Protocol for Collecting Attendance and Content Coverage Data (Instrument 5)

The data collected from these instruments will provide a detailed understanding of program implementation in the three rigorous impact study sites.

B1. Respondent Universe and Sampling Methods

OAH has selected two program sites to participate in an experimental impact study, one of which is a current PAF grantee. OAH has selected a third program site to participate in a quasi-experimental design evaluation. The sites are not meant to be representative of PAF-funded programs, or all programs for expectant and parenting youth.. Site selection has focused on programs that (1) are large enough to support an impact study, (2) are implementing programs in a way that is amenable to random assignment or a quasi-experimental design, and (3) address priority gaps in the existing research literature on evidence-based approaches to assist pregnant and parenting youth.

The implementation data collection will take place in each of the three sites. Within each site, interviews will be conducted with diverse staff and community members who play substantive roles in program implementation, who are knowledgeable about the origins and operations of their program, and who can discuss any challenges encountered and how they were resolved. In addition, all frontline staff and supervisors will be asked to complete a staff survey about program implementation and the support they receive for it. Finally, in all sites, focus groups or individual interviews will be held with participating youth who agree to participate in a focus group discussion and have parental consent to do so.

There are three sites participating in the PAF Study. Two of these sites (California and Texas) will use an experimental design and primary data collection through surveys of youth. The third site, in Washington, DC, will use a quasi-experimental design and rely on administrative data provided through data use agreements with three local public agencies. Youth will not be surveyed; however, the site will participate in data collection for the in-depth implementation study.

The three study sites are described in detail in Appendix A, Overview of the PAF Study

B2. Procedures for Collection of Information

Site Visits for the Implementation Study. We expect that the first site visit will occur in the first year that implementation begins and will focus on input from program leadership and understanding the lessons learned from early implementation experiences. The remaining visit(s) will be scheduled and conducted during the subsequent two years, and we will observe implementation at multiple programming locations within each site. The exact timing will depend on the schedule of program activities.

The master list of topics and questions for staff interviews during site visits (Instruments 1 and 2) identify the information that will be gathered from program leaders, supervisors, and front-line staff to document the program context, implementation plans, the implementing organization and partner organizations, implementation systems, youth participation and engagement, and actual service delivery. Preparation for site visits will involve using the master list of topics to develop discussion guides customized to each site to ensure that site visitors collect the relevant needed information in an efficient, consistent way from the appropriate respondents.

Staff Survey for the Implementation Study. The study team will survey all of the front line staff (home visitors, school-based coordinators, case managers) delivering the programs; the appropriate supervisors in each of the subsites; and, in California, all program facilitators delivering the business-as-usual services to control group youth. We anticipate that about 105 respondents will take the survey (Instrument 3) across all sites. The 35-minute survey will be in pencil and paper format and designed to capture targeted input from all of the staff implementing the program since in-person interviews would not be practical or feasible across the large number of site locations and agencies. The survey uses closed- and open-ended questions aligned with and designed provide the data for the planned analyses. We anticipate that the survey will be administered during the final site visit in each site.

Participant Focus Groups and Interviews for the Implementation Analysis. Focus groups or interviews (Instrument 4) will be conducted with a subset of program participants during site visits. The objective of the focus groupsdiscussions will be to explore participants’ perspectives on the availability, quality, and value of program services. The focus groupsdiscussions will be used to learn about participants’ motivations for enrolling in the program, their participation and response to incentives offered, their experiences with each of the core services offered, their perceptions of the benefits of participation, and their overall satisfaction with program services.

Site visitors will work with local staff to arrange the groups or interviews at convenient times and locations and to recruit. For focus groups ofwill will work with staff to invite 8 to10 program participants and control group youth from multiple program locations where appropriate . We will work with site staff to offer concrete assistance in the form of child care, food, transportation, and other potential incentives because these will be important when recruiting expectant or parenting teen participants. If it is infeasible for youth to travel to a focus group, we will invite them to share their experiences through an interview.

Program Attendance and Content Coverage Data for the Implementation Study. Program attendance and content coverage data will be collected for every participant at each of the two random assignment sites. If the site already maintains this attendance information in its existing administrative records system, sites will extract these data from their administrative system and provide it to the evaluation team. Attendance and content coverage data will allow the evaluation team to document the proportion of program services that was actually delivered to participants., and the extent to which there is a contrast with the control group’s program services.

B3. Methods to Maximize Response Rates and Deal with Non-Response

To ensure high response rates to data collection efforts associated with the implementation study, site visits will be planned well in advance so that all identified respondents can participate in individual or group interviews, as appropriate. To increase participation in focus groups and interviews, youth who volunteer to participate will be offered a $25 gift card for their time, as is customary in other federal evaluations such as PREP. We will also work with the program sites to offer child care to improve participation. To ensure that attendance and program content coverage data are recorded completely and accurately, the evaluation team will routinely review these data provided by sites and follow up with program staff if information is incomplete.

B4. Test of Procedures or Methods to be Undertaken

The information collection instruments are similar to discussion protocols that have been used successfully in prior studies, such as the PREP (ACF) and PPA evaluations (OAH).

Attachment E provides a question-by-question source table for the staff survey. Many of the items are drawn from established sources and have been tested and refined through those survey efforts.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The PAF evaluation Implementation Study site visits will be conducted by OAH’s contracting organization, Mathematica Policy Research. Mathematica will also conduct all analyses of the data. Attachment B lists the individuals whom ACF consulted on the collection of the implementation study instruments.