Part B - Clean version - 3-5-12[1]

Supporting Justification for OMB Clearance of the Personal Responsibility Education Program (PREP) Multi-Component Evaluation

OMB Control #0970-0398

Part B: Statistical Methods for Design Survey Data Collection

November 2011

Submitted By:

U.S. Department of Health and Human Services

Administration for Children and Families

Office of Planning, Research and Evaluation

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officers:

Dirk Butler

Clare DiSalvo

Introduction

The Patient Protection and Affordable Care Act, signed into law in March of 2010, established the Personal Responsibility Education Program (PREP) which funds programs designed to educate adolescents on both abstinence and contraception for the prevention of pregnancy and sexually transmitted infections, including HIV/AIDS, and at least three adulthood preparation subjects. ¹ PREP provides $55.25 million in formula grants to States to “replicate evidence-based effective program models or substantially incorporate elements of effective programs that have been proven on the basis of scientific research to change behavior, which means delaying sexual activity, increasing condom or contraceptive use for sexually active youth, or reducing pregnancy among youth.” The PREP legislation also requires an evaluation.

The goal of the PREP Multi-Component Evaluation will be to document how programs funded through the State PREP program are designed and implemented in the field and to assess selected PREP-funded programs’ effectiveness. The project will include three primary, interconnected components, each of which is a study in its own right. These components are:

1. a Design and Implementation Study (DIS): a broad descriptive analysis of how states designed and implemented PREP programs,

2. a Performance Analysis Study (PAS): the collection and analysis of performance management data, and

3. an Impact and In-depth Implementation Study (IIS): impact and in-depth implementation evaluations of four to five specific PREP-funded sites.

As part of the first component of this project – the “Design and Implementation Study,” the broad descriptive study of how states designed and implemented PREP programs – ACF now seeks approval for a “Design Survey” data collection instrument. The purpose of the “Design Survey” data collection effort is to conduct semi-structured interviews with administrators in each of the states that received PREP state grants in order to better understand what key decisions states made regarding the design of their PREP-funded programs and why they made those decisions.

B1. Respondent Universe and Sampling Methods

45 states and the District of Columbia received FY 2010 PREP funds. We expect two respondents from each state to be contacted and interviewed over the course of two years. Respondents will be state-level PREP program administrators. The specific individuals to be interviewed will be identified through a review of the applications submitted by states for PREP funds and conversations with federal staff.

B2. Procedures for Collection of Information

The data collection modality will be interviews. Interviews will primarily be conducted by phone; however, a few in-person interviews may be conducted if doing so would promote an efficient use of contract resources. (For example, in-person interviews may be conducted if the research team is already on site for another data collection effort related to the PREP Evaluation.)

ACF has provided the contractor with a key contact for each state and the District of Columbia that received PREP funding. The contractor will send an email to the key contact that describes the “Design and Implementation Study”, and the purpose of and nature of the questions on the “Design Survey” for which an interview is being requested. Attached to that email will be a document that provides an overview of the entire PREP evaluation, including the “Design and Implementation Study” component. The email will ask the intended respondent to select a day and time for an interview that will not last longer than one hour. The email will also suggest that the key contact recommend a respondent, if the key contact does not believe they are in the best position to address the “Design Survey” questions. Telephone follow-up will be initiated if the key contact does not respond within three business days.

The specific questions asked during each interview will vary, depending on 1) what is already known about the respondent’s program design decisions (e.g. we will not ask questions for which we already have answers, based on the contractor’s prior review of program documents and administrative data) and 2) the discretion of the interviewer, who will adapt his or her questions based on the respondent’s answers, while still touching on key themes across interviews.

The “Design Survey” instrument, therefore, will serve as a pool of possible questions which will be drawn upon by the contractor to guide informal, semi-structured interviews (i.e. they are not rigidly structured, systematic surveys).  Each interviewer will use a sub-set of the questions listed in the interview guide and each interview will last for no more than 1 hour.  A specific protocol will be developed for each interview in advance of the call. This overall approach has the benefit of reducing burden for respondents, because each interview will be tailored for the specific respondent who is being interviewed.  This strategy has worked very successfully in past data collection efforts like this (e.g. with implementation studies). 

Contractor staff will takes notes during discussions, obtain relevant written materials that are readily available, and prepare written summaries of each discussion and group of discussions for submission to ACF. There will be no formal consent process for the interviews, as the contractor’s IRB of record has waived both verbal and written consent.

B3. Methods to Maximize Response Rates and Deal with Non-response

We expect to obtain a very high response rate (95 percent) among selected respondents. Several factors will help ensure a high rate of cooperation among respondents. First, grantees are required to participate in evaluation activities per the requirements of their grant award.² Individuals may opt out of these interviews; however, state agencies that received PREP funds are expected to make a state official available who can complete the interview. Second, the stakeholders who will be interviewed or surveyed are all heavily invested in the issues surrounding teen pregnancy prevention and thus should be motivated to participate in the interview. Third, federal staff (PREP project officers) will contact state agencies who do not respond to interview requests and ask them to make an official available who can participate.

B4. Test of Procedures or Methods to be Undertaken

The information collection instruments are similar to discussion protocols that have been used successfully in prior studies.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information collection for this component of the evaluation will be done by Mathematica Policy Research, the contractor for this study, on behalf of ACF.

Key input to the current data collection instruments was received from the following individuals:

Seth Chamberlain

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

(202) 260-2242

Caryn Blitz

Office of the Commissioner

Administration on Children, Youth & Families

Administration for Children and Families

U.S. Department of Health & Human Services

(202) 401-9225

Marc Clark

Division of Teen Pregnancy Prevention Programs

Family and Youth Services Bureau

Administration for Children and Families

Administration for Children and Families

U.S. Department of Health & Human Services

(202) 205-8496

Lisa Trivits

Division of Children and Youth Policy

Office of the Assistant Secretary for Planning and Evaluation

U.S. Department of Health and Human Services

(202) 205-5750

Diana Tyson

Division of Children and Youth Policy

Office of the Assistant Secretary for Planning and Evaluation

U.S. Department of Health and Human Services

(202) 401-6670

Amy Farb

Office of Adolescent Health

Office of the Assistant Secretary for Health

U.S. Department of Health and Human Services

(240) 453-2836

1 PREP legislation outlines the six adulthood preparation subjects:

(i) Healthy relationships, Including, marriage and family interactions.

(ii) Adolescent development, such as the development of healthy attitudes and values about adolescent growth and development, body image, racial and ethnic diversity, and other related subjects.

(iii) Financial literacy.

(iv) Parent-child communication.

(v) Educational and career success, such as developing skills for employment preparation, job seeking, independent living, financial self-sufficiency, and workplace productivity.

(vi) Healthy life skills, such as goal-setting, decision making, negotiation, communication and interpersonal skills, and stress management.

2 See the State Personal Responsibility Education Program (PREP) Funding Opportunity Announcement (p. 11), available at http://www.acf.hhs.gov/grants/open/foa/view/HHS-2010-ACF-ACYF-PREP-0125.