Part B

Supporting Justification for OMB Clearance of the Personal Responsibility Education Program (PREP) Multi-Component Evaluation

Part B: Statistical Methods for Field Data Collection

November 2011

Submitted By:

U.S. Department of Health and Human Services

Administration for Children and Families

Office of Planning, Research and Evaluation

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officers:

Dirk Butler

Clare DiSalvo

Introduction

The Patient Protection and Affordable Care Act, signed into law in March of 2010, established the Personal Responsibility Education Program (PREP) which funds programs designed to educate adolescents on both abstinence and contraception for the prevention of pregnancy and sexually transmitted infections, including HIV/AIDS, and at least three adulthood preparation subjects.  PREP provides $55.25 million in formula grants to States to “replicate evidence-based effective program models or substantially incorporate elements of effective programs that have been proven on the basis of scientific research to change behavior, which means delaying sexual activity, increasing condom or contraceptive use for sexually active youth, or reducing pregnancy among youth.”

The goal of the PREP Multi-Component Evaluation will be to document how programs funded through the State PREP program are designed and implemented in the field and to assess selected PREP-funded programs’ effectiveness.  The project will include three primary, interconnected components, each of which is a study in its own right. These components are:

1. a Design and Implementation Study (DIS): a broad descriptive analysis of how States designed and implemented PREP programs,

2. a Performance Analysis Study (PAS): the collection and analysis of performance management data, and

3. an Impact and In-depth Implementation Study (IIS): impact and in-depth implementation evaluations of four to five specific PREP-funded sites.

As part of the third component, ACF now seeks approval for field data collection instruments. The purpose of the field data collection effort is to identify potential sites for inclusion in the “Impact and Implementation Study,” which entails random assignment evaluations and in-depth implementation evaluation in 4-5 specific sites.

All the measures in the instruments included in this ICR were originally approved under OMB Clearance No. 0970-0360 as part of the Evaluation of Adolescent Pregnancy Prevention Approaches (PPA) coordinated by the Office of Adolescent Health (OAH). ACF will continue to coordinate PREP data collection instrument development with OAH and other offices across HHS that oversee teen pregnancy prevention programming and evaluation (e.g. the HHS Assistant Secretary for Planning and Evaluation (ASPE) and CDC’s Division of Reproductive Health (CDC/DRH)).

B1. Respondent Universe and Sampling Methods

The purpose of this information collection is to help ACF identify and select 4-5 PREP-funded teen pregnancy prevention intervention programs for inclusion in the “Impact and In-Depth Implementation Study,” which entails random assignment impact evaluation and in-depth implementation evaluation in 4-5 sites.

Site to be included in this study will be selected based on the following criteria: 1) the extent to which a site could support a random assignment impact evaluation (for example, whether the site could generate a sufficiently large sample size and whether there is a strong treatment-counterfactual distinction) and 2) whether the inclusion of the site in the evaluation would address ACF’s key research questions (whether it would test a program model designed to serve a vulnerable population of special interest to ACF, such as foster youth or run-away and homeless youth).

As background, it is important to note that while PREP state grant funds are awarded to and administered by states, the sites selected for impact evaluation will most likely be sub-awardees. Most states are distributing their state grant funding to a number of community-level sub-awardees within their state via a competitive grant process. These sub-awardees may be county health departments, school districts, or local community organizations, for example. Each of these sub-awardees is then responsible for implementing their own PREP-funded teen pregnancy prevention program. While we are requesting a small amount of burden in order to be able to speak with state-level PREP administrators, the bulk of the burden that we are requesting for the field instrument is to speak with sub-awardee-level respondents.

46 states and the District of Columbia received PREP state grant funds. For the field data collection effort, of this total, we plan to reach out to up to 10 states in order to identify 4-5 sites for the “Impact and In-Depth Implementation Study”. These states will be identified through a review of documents available to ACF and discussions with federal staff. Within each state, we will speak to up to 1 macro-level coordinator, 2 program directors, 4 program staff, and 7 school administrators.

In other words, the respondent universe for this data collection effort is made up of the following individuals:

• Macro-Level Coordinators—state-level PREP coordinators or other state-level coordinators;

• Practitioners—directors or staff of PREP-funded pregnancy prevention programs, including school- and community-based programs as well as local and state agencies, as appropriate;

• As appropriate, school administrators or individuals who coordinate, oversee, or otherwise work with PREP-funded programs within educational settings.

B2. Procedures for Collection of Information

Informal, semi-structured discussions/interviews will be conducted by telephone and in-person. Discussions will be conducted by senior members of the evaluation contractor team. Contractor staff will use the approved instruments to guide discussions, to ensure that appropriate topics are covered, given the type of stakeholder being interviewed. Contractor staff will takes notes during discussions, obtain relevant written materials that are readily available, and prepare written summaries of each discussion for submission to ACF.

Discussions will be conducted commencing with the receipt of OMB clearance, and will occur most heavily in the first year of the clearance but may extend into the second and third years.

How the number of sites (4-5) for the “Impact and Implementation Study” was calculated: The goals of the PREP Multi-Component Evaluation are to document the design and implementation of PREP programs, collect performance measure data for PREP programs, and to assess the effectiveness of a small number of PREP-funded programs. The three components of the evaluation – the “Design and Implementation Study,” the “Performance Analysis Study,” and the “Impact and Implementation Study” - are designed to meet these three goals. ACF has determined that, with the amount of funding available for the project, we are able to include 4-5 sites in the impact evaluation component of the project.

B3. Methods to Maximize Response Rates and Deal with Non-response

The purpose of this effort is not to collect data for statistical analysis. Rather, it is to identity sites for inclusion in a random assignment impact evaluation. While states are required to participate in the evaluation if selected, it is essential that for each site selected, key local stakeholders be invested in the project and eager to participate.

B4. Test of Procedures or Methods to be Undertaken

As discussed in Section A12, the discussion guides were pre-tested with ACF staff and the burden was estimated from these tests. The information collection instruments are similar to discussion protocols that have been used successfully in prior studies.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for the Field Data Collection will be collected by the evaluation contractors, and their subcontractors, on behalf of ACF. With oversight from the Administration for Children and Families (ACF), the contractor will be responsible for the finalizing the study design, data collection, analysis, and report preparation. Key input to the field data collection instruments was received from the following individuals:

Department of Health and Human Services

Administration for Children and Families

Family and Youth Services Bureau

• Dirk Butler, Research Analyst

Office of Planning, Research, and Evaluation

• Naomi Goldstein, Director

• Nancye Campbell, Senior Research Analyst

• Seth Chamberlain, Research Analyst

• Clare DiSalvo, Presidential Management Fellow

Office of the Assistant Secretary for Planning and Evaluation

• Lisa Trivits, Research Analyst

Inquiries regarding statistical aspects of the study design should be directed to:

Clare DiSalvo

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

370 L'Enfant Promenade, S.W.

Washington, DC 20477

202-401-4537

Ms. DiSalvo is a project officer, along with Dirk Butler of ACYF, on the PREP Multi-Component Evaluation.