June 2014
Submitted by:
Office of Adolescent Health
U.S. Department of Health and Human Services
1101 Wootton Parkway, Suite 700
Rockville, MD 20852
Project Officer: Amy Farb
The Office of Adolescent Health (OAH) Teen Pregnancy Prevention (TPP) grant program was launched in spring 2010 as a key early piece of the federal government’s ongoing “evidence and innovation” agenda. The program provides approximately $100 million in annual competitive contracts and grants to public and private entities to fund medically accurate and age appropriate programs to reduce teen pregnancy. The program features a “tiered-evidence” grant design that reserves most of the funding for grants to replicate programs with existing evidence of effectiveness (Tier 1). A smaller proportion of funding is reserved to encourage innovation in the field by implementing and rigorously testing promising new programmatic approaches (Tier 2). The first Tier 1 Replication grants were awarded in fall 2010 to 75 state or local organizations, for programming to start in fall 2011.
Consistent with the program’s focus on evidence, OAH has undertaken a range of evaluation activities associated with the TPP program. Related to Tier 1 of the program, nine grantees are participating in the ongoing federal TPP Replication Study1, a large-scale, multi-site random assignment evaluation of three different evidence-based teen pregnancy prevention programs. Other grantees are using a portion of their funding to conduct their own “local” program impact and implementation evaluations. All grantees collect data on a uniform set of performance measures2 and report them to OAH on a semi-annual basis through an online system. The burden associated with these ongoing data collection activities has been previously reviewed and approved by OMB.
With this information collection request, OAH seeks approval for additional data collection instruments, to conduct a complimentary cost study of selected TPP grantees. The proposed cost study adds a new and unique contribution to OAH’s portfolio of evaluation activities. The study has three main components: (1) a cost analysis to determine the cost of implementing select evidence-based teen pregnancy prevention programs; (2) an economic evaluation to determine the economic impact of select evidence-based teen pregnancy prevention programs; and (3) the development of guidance and tools for OAH to use to collect and analyze cost data from potential future grantees in a systematic, standardized way. OAH has contracted with Mathematica Policy Research to conduct this three-year (2013-2016) study.
This information collection request focuses specifically on data collection instruments for the first two study components: (1) the cost analysis and (2) the economic evaluation. Any data collection instruments developed in support of the third study component (guidance and tools for use with future grantees) will be submitted separately in future years of the study.
a. Universe
The cost analysis will be conducted with a sample of up to 30 grantees currently receiving funds to replicate evidence-based teen pregnancy prevention programs under Tier 1 of the OAH TPP grant program. There are roughly 75 total grantees receiving funding through this component of the TPP program. Thus, the projected sample for this study represents just under half of the total number of grants awarded.
As discussed below, about half of the 30 grantees selected for the cost analysis will also participate in the economic evaluation component of the study. This smaller sample is necessary because the economic evaluation requires the availability of program impact estimates, and not all of the grantees are conducting the program impact evaluations needed to produce these estimates.
The study is not intended to select a representative sample of all grantees. Rather, the site selection process will use “judgmental” or “purposive” sampling techniques designed to answer particular research questions or meet certain analytic needs. In particular, sites will be selected accounting for the following factors:
Number of different programs being implemented. Some grantees are using their funds to implement more than one teen pregnancy prevention program, whereas others are implementing a single program. Grantees implementing more than one program will be excluded from consideration unless they can accurately apportion costs being used to support each individual program.
Availability of program impact estimates. Only a limited number of grantees are conducting impact evaluations of their programs. Because program impact estimates are necessary for the planned economic evaluation, those grantees that are conducting evaluations will be prioritized for site selection.
Number of grantees implementing the same program. To develop hypotheses about the factors that may be driving program costs, it is necessary to compare cost estimates across different grantees implementing the same program. Such comparisons will help facilitate analysis of why costs may be higher for some grantees than others, net of any costs common to the same program. To allow for these comparisons, site selection will focus on grantees that are implementing common programs. The study sample may thus comprise 30 different grantees but only 8 or 9 different TPP program models.
c. Selection of Respondents for Each Grantee
The data collection instruments may require responses from several different individuals within each of the 30 participating grantees. The Cost Tool (Instrument #1) and Implementation Tool (Instrument #2) will be administered to the grant director or other person most knowledgeable about the program finances and implementation. The Staff Time-Use Survey (Instrument #3) will be administered to the larger group of staff responsible for planning and implementing the program. The Economic Evaluation Form (Instrument #4) will be administered to the evaluation lead or other person most knowledgeable about findings from the program impact study. A more detailed description of these instruments and responses can be found under B2.
The data collection will involve four separate instruments: (1) Cost Tool; (2) Implementation Tool; (3) Staff-Time Use Survey; and (4) Economic Evaluation Form. The data collection procedure will vary for each instrument depending on the specific purpose.
Cost Tool. For the core cost data collection, one respondent in each of the 30 grantee sites will be asked to provide detailed information on program costs by populating an electronic spread-sheet based instrument (Instrument #1). The spreadsheets will collect cost information on staff salaries and fringe benefit rates; subcontracted service providers or other consultants; materials, supplies, and equipment purchased for the program; training for program staff; intervention materials purchased for facilitators and/or participants to use; licensing or other fees paid to program developers, facilities used to house program staff or delivery program activities, and indirect or shared costs for the grantee agency. The instrument is self-administered and includes detailed instructions for completing, though respondents will also be given the phone number and e-mail address of a study team liaison to contact with any questions.
Implementation Tool. In order to assess the resources used by a program, it is important to have detailed information about program implementation. This tool (Instrument #2) is designed to obtain background information about each grantee’s program, to provide information on the types of resources that are most important in the program. To minimize the level of burden on sites, the study team will pre-populate the instrument with any program information currently known by OAH or publically available. The pre-populated information will then be reviewed by telephone with the grantee lead or other designated respondent using a semi-structured interview format, in which the cost study team, not the grantee, will be responsible for completing the information. The grantee will later have a chance to review and edit or comment on the information provided, as appropriate.
Staff Time Use Survey. The staff time use survey will be administered as a web survey (Instrument #3) to be completed by all staff who spend time delivering program services, supporting service delivery, or administering activities associated with the grant, including consultants or contractors. The activity log will ask staff to estimate how they spent their time during a typical week in the past month, using codes and categories for commonly conducted activities, such as teaching a class or facilitating a small-group session. Staff will be asked to estimate the percentage of time they spend on the program being implemented, and within that time, how much time they spend in various activity categories. This information on staff time allocation will be important for understanding how program costs are allocated across program activities and for developing estimates of the unit costs of providing specific services. The survey is estimated to take no more than 20 minutes and can be completed at any time the respondent finds convenient, reducing respondent burden. The survey will be administered to each staff member up to two times during the data collection period. If some sites have a very large number of staff involved (for example, more than 20), we will select a random sample of staff in key job categories to complete the survey.
Economic Evaluation Form. The grantees involved in the economic evaluation will need to designate one respondent to provide information on the grantee’s impact evaluation. The study team will contact this designated response to complete the Economic Evaluation Form (Instrument #4), which will collect information needed to generate program cost-effectiveness estimates for each site. To minimize the level of burden on sites, the study team will pre-populate the instrument with any program information currently known by OAH about the ongoing evaluations. The pre-populated information will then be reviewed by telephone with the designated respondent using a semi-structured interview format, in which the cost study team, not the respondent, will be responsible for completing the information.
The Cost Study of Evidence-Based TPP Programs is collecting data from grantees funded by OAH, with the understanding that they will cooperate with OAH evaluations of the grant programs. In addition, the cost study team will work with OAH to build support for the study among grantees, develop clear MOUs which lay out the obligations of both sides in the relationship, and minimize burden during data collection by use of a web survey that can be complete any time and any place, as well as less intrusive e-mail contacts. Mathematica has extensive experience in collecting cost data and web-based data collection with high response rates from staff in education, social services, and health programs. However, even in the rare event of nonresponse, the study findings will not be significantly affected, because the sites are not intended to be a representative probability sample of all OAH grantees.
B4. Tests of Procedures or Methods to be Undertaken
To ensure that the study data collection instruments meet their intended purpose and impose minimal burden on respondents, OAH is conducting a pilot testing with two current TPP grantees: (1) Inspira Health Network in Vineland, New Jersey, and (2) the social-service agency Women Accepting Responsibility in Baltimore, Maryland. OAH will report to OMB on the results of the pilot test.
The agency official responsible for receiving and approving contract deliverables is:
Amy
Farb
240-453-2836
[email protected]
Office
of Adolescent Health/DHHS
1101 Wootton Parkway, Suite
700
Rockville, MD 20852
The persons with primary responsibility for designing and administering the information collection are:
Andrew Burwick
609-945-6566
Mathematica Policy Research
920 Mapleton Ave.
Boulder, CO 80304
Brian Goesling
609-945-3355
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543-2393
Anne Gordon
609-275-2318
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543-2393
Heather Zaveri
202-264-3441
Mathematica Policy Research
1100 1st Street, NE
Washington, DC 20002-4221
1 OMB approval number 0990-0394
2 OMB approval number 0990-0392
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Justification for OMB Clearance of Evaluation of Pregnancy Prevention Approaches |
| Author | Barbara Collette |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |