0990-Performance Measures_Part_A_Supporting_Statement_10 21 15 Clean

Supporting Justification for the OMB Clearance of the

“Office of Adolescent Health Teen Pregnancy Prevention, FY 2015-2020 Performance Measure Collection”

Part A: Justification for the Collection of the Data

May 2015

Submitted by:

Office of Adolescent Health

U.S. Department of Health and Human Services

1101 Wootton Parkway, Suite 700

Rockville, MD 20852

Project Officer: Amy Farb

Background

The Office of Adolescent Health (OAH) is requesting approval for the collection of performance measures for the second cohort of grantees to be awarded Teen Pregnancy Prevention (TPP) Program funding. The TPP Program was initiated in FY 2010 as one of six major evidence-based policy initiatives across the Federal government. OAH supports two types of grants through the TPP program (1) projects that replicate evidence-based TPP program models that have been shown to be effective through rigorous evaluation, referred to as Tier 1 and (2) research and demonstration projects that develop and test additional models and innovative strategies to prevent teen pregnancy, referred to as Tier 2. Funding for the TPP Program is authorized under the Consolidated Appropriations Act, 2014 (P.L. 113-76) and the Continuing Resolution for FY 2015 (Public Law 113-164). The Act provides $100 million in FY 2015 for making competitive awards to public and private entities to fund programs that reduce teen pregnancy. In FY 2015, OAH anticipates awarding new 5-year cooperative agreements.

Performance measures have consistently been a requirement of the TPP grant program since its inception. The majority of the measures that are included in this information collection request previously received OMB clearance (0990-0392) and have been successfully used over the past 4 years with the initial cohort of grantees. Based on our data collection experiences over the last four years, and in consideration of the goals of the second cohort of grantees, the proposed measures have been modified slightly, as presented later in this document. Based on our experience, we have removed some questions with poor response rates and added a few that are important for tracking the progress of the new grants. This Supporting Statement requests approval for collection of performance measures for the second cohort of TPP grantees.

In FY2015, OAH awarded TPP grants in 5 categories. The majority of the funding is available for Tier 1 grants, a small amount going to intermediary organizations to provide capacity building assistance to youth-serving organizations that replicate evidence-based TPP programs in high-need communities (Tier 1A) and the remainder for organizations that will replicate evidence-based TPP programs to scale in at least 3 settings in communities with the greatest need (Tier 1B). Tier 2 grants will be awarded to intermediaries to support early stage innovation in teen pregnancy prevention (Tier 2A), organizations that will rigorously evaluate new or innovative approaches for preventing teen pregnancy (Tier 2B), and in partnership with the Centers for Disease and Control and Prevention (CDC), organizations implementing rigorous evaluations of innovative interventions to prevent teen pregnancy designed for young men (Tier 2C). More detailed information, including the number of awards in each category may be found in Exhibit 1.

The performance measures data collection will fulfill several important functions. First, OAH expects its grantees to utilize the measures to make continuous quality improvement in their program implementation and inform their partners and stakeholders about implementation and sustainability progress. Second, performance measures provide OAH with metrics for monitoring TPP grantees so that project officers can provide technical assistance when needed. Finally, OAH uses the measures to report to OASH, OS, our budget office, and Congress on the grantees’ progress. Performance Measure data collection is a requirement of all TPP grant awards and is included in the funding announcements.

Exhibit 1: New Cohort of TPP Projects to be Funded in FY 2015

New TPP Project Funding Opportunity in FY 2015	Anticipated Total Annual Funding	Estimated # of Awards	Estimated Annual Award Amount	Rigorous Evaluation
Capacity Building to Support Replication of Evidence-Based TPP Programs (Tier 1A)	$5 million	8	$400,000 - $750,000	No
Replicating Evidence-Based TPP Programs to Scale in Communities with the Greatest Need (Tier 1B)	$60 million	50	$500,000 - $2,000,000	No
Supporting and Enabling Early Innovation to Advance Adolescent Health and Prevent Teen Pregnancy (Tier 2A)	$3 million	2	$1,000,000 -$1,500,000	No
Rigorous Evaluation of New or Innovative Approaches to Prevent Teen Pregnancy (Tier 2B)	$18 million	21	$400,000 - $1,000,000	Yes
Effectiveness of TPP Programs Designed Specifically for Young Males (Tier 2C)	$2 million	3	$600,000 - $1,000,000	Yes

A1. Need and Legal Basis

The performance measure collection is critical to OAH because it provides the agency with data to both effectively manage the TPP program, and to comply with accountability and federal performance requirements for the 1993 Government Performance and Results Act (P.L. 103-62). Moreover, collecting and reporting on data for performance measures are funding requirements for the grants, as stated in the funding opportunity announcement.

The performance measures to be reported to OAH are summarized in Exhibit 2. The specific questions that grantees will answer to address these measures are found in Appendix C.

Exhibit 2: Performance Measures for the FY 2015-2020 Cohort

Performance Measure Constructs	Data source
Grantee-Level Measures
Dissemination	Grantee/Sub-awardee Administrative Records
# of manuscripts accepted for publication or published in journals
# of national, regional, or state-level presentations
Number of Partners
# of formal and informal partners	Grantee/Sub-awardee Administrative Records
Retention of partners
Training	Grantee/Sub-awardee Administrative Records
# of new facilitators trained
# of facilitators receiving follow-up training
Linkages to Youth Friendly Health Care1
# of referrals made by program staff to youth-friendly off-site providers, by type of services	Grantee/Sub-awardee Administrative Records
Sustainability1
Amount of new funding received to support the program and services; number and type of implementation partners; dissemination efforts	Grantee/Sub-awardee Administrative Records
Program-Level Measures
Participant Reach
# of youth served, by characteristics (e.g., age, gender, race/ethnicity,)	Grantee/Sub-awardee Administrative Records
# of parents/other clients served
Dosage	Grantee/Sub-awardee Attendance Records
Mean and median percent of total intended program services received by youth
% of youth who received > 75% of the program
Fidelity
Adherence to number of program-specified sessions	Fidelity Monitoring Logs
Adherence to program-specified activities, based on facilitator self-assessment
Adherence to program-specified activities, based on observations	Observation Forms
Quality of implementation
Cost1
Cost of implementing the project	Grantee/Sub-awardee Administrative Records
Cost of providing each program service

A.2 Information Users

The data collection activities will provide information to OAH leadership and program officers to help them to more effectively manage the TPP program. We anticipate that the data will be made available to Congress by the Office of Management and Budget (reach, dosage, training and partners), and the public at large (all measures) to assess program performance. Use of these data is vital for ensuring on-going improvement of the TPP program and through dissemination efforts, broader understanding and support of programs designed to prevent teen pregnancy.

A.3. Use of Information Technology and Burden Reduction

Grantees will enter performance measure data into a multi-use, Web-based reporting system (the same system created for the initial grant cohort), either by using point and click entry or by uploading spreadsheets using a template. The Web-based system can reduce burden for respondents by programming in skip patterns, so that grantees only have to look at questions or uploading data that are relevant for them. Programming automatically performs necessary calculations for respondents, and will validate responses. For point and click entry, a branching mode of presentation allows respondents to go directly to the sections they need, without having to go through the system in a linear progression. The system also automatically produces a data set of measures across all grantees using relevant filters (e.g., for Tier 2B grantees only), which saves time on preparation of the data for analysis. Data are also available for grantees to export into Excel, CSV, or Word files to custom design reports. Screenshots of the current web-based reporting system are included in Appendix E.

A.4. Efforts to Identify Duplication and Use of Similar Information

The OAH performance measures data collection is the only data collection that will provide information on the performance of the TPP program. The data collection will make use of existing data to the extent possible. The performance measures are collected by grantees as part of their routine administrative records (e.g., numbers of publications, numbers of participants, and attendance at program sessions). Please see Exhibit 2 for data sources for each performance measure.

With three exceptions, all of the constructs, measures, and questions in the proposed collection have been previously approved by OMB through collection OS 0990-0392, “Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection.” The three new constructs are those that report on linkages to youth-friendly health care, sustainability, and cost (see Appendix C for the complete list of measures).

Please note that a renewal Information Collection Request was submitted on 1/30/2015 to cover the final data collection in Fall 2015 for OMB # 0990-0392 “Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection.” The current request is new due to the new sample, new burden, and new questions.

A.5. Impact on Small Businesses

No small businesses will be involved in the collection of data in this study.

A.6. Consequences of Not Collecting the Information/Collecting Less Frequently

GPRA requires that government agencies report on their performance measures annually. Therefore, it is essential that grantees report on these performance measures at least annually to OAH. In addition, collection and reporting of performance measure data is a requirement of all TPP grantees as stated in the Funding Opportunity Announcement.

A.7. Special Circumstances

There are no special circumstances that occur when collecting this information.

A.8. Federal Register Comments and Persons Consulted Outside the Office of Adolescent Health

A 60-day notice was published in the Federal Register on February 24, 2015, in Volume 80, Number 36, page 9723, and provided a 60-day period for public comments (Appendix A). No public comments were received.

In 2010 and 2011, OAH consulted with staff of RTI International, which was the contractor responsible for assisting OAH in developing the performance measures and performance measure reporting system, and a panel consisting of experts in the fields of performance measurement, teen pregnancy prevention, and evidence-based practice. In addition, OAH presented information on the performance measures to TPP grantees and their evaluators at two conferences, and solicited their input. OAH also consulted and received feedback from other Federal staff working in the area of teen pregnancy prevention from ASPE, ACF, and CDC. The bulk of the performance measures that are part of this clearance package have been successfully used during the past 4 years in the data collection for the “Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection.”

A list of individuals in the expert panel who provided input regarding the process evaluation is found in Exhibit 3.

Exhibit 3. Persons Consulted Outside the Agency for Performance Measure Collection for FY 2010 Grantees

Expert Work Group

Donald Moynihan

[email protected]

University of Wisconsin

Lafollette School of Public Affairs

305 Observatory Hill Office Building

1225 Observatory Dr

Madison, WI 53706

(608) 263-6633

Kathryn Newcomer

[email protected]

George Washington

SPPPA

MPA Bldg 601

805 21st St NW

Washington, DC 20052

(202) 994-3959

Katherine Suellentrop
[email protected]
The National Campaign to Prevent Teen and Unplanned Pregnancy
1776 Massachusetts Ave, NW, suite 200
Washington, DC 20036
(202) 478-8515

Edward Mullen

[email protected]

Columbia University

School of Social Work

1255 Amsterdam Ave Room 1102

New York, NY 10027

(212) 851 2413

Douglas Kirby (deceased)

[email protected]

ETR Associates

4 Carbonero Way

Scotts Valley, CA 95066

(831) 438-4060

Forrest Alton

[email protected]

SC Campaign to Prevent Teen Pregnancy

1331 Elmwood Avenue, Suite 140

Columbia, SC 29201

(803) 771-7700

Emily Ball

[email protected]

Administration for Children and Families

26 Federal Plaza

New York, NY 10278

(212) 264-2890 x273

A.9. Payments to Respondents

There are no payments to staff of grantee organizations completing the performance measure reporting form.

A.10. Assurance of Confidentiality

All data are reported de-identified. Grantees create their own unique randomly assigned identification numbers prior to entering any data.

The Web-based reporting system has been designed to ensure the security of the data obtained. Electronic data are stored in a location within the RTI network that provides the appropriate level of security based on the sensitivity or identifiability of the data. No personal identifiers will be used in the reporting of any data.

Individual users designated by the grantees will be assigned user names and passwords that will grant them access to the project website. There, users will have the opportunity to provide data that will be stored in a secure Microsoft SQL Server database utilizing a relational table structure, facilitating expedient data retrieval and analysis. The database server, located at RTI, will be accessible only to the statisticians and analysts assigned to this project. Electronic communications will occur via a secure Internet connection. All transmissions will be encrypted with 128-bit encryption through secure socket layers (SSL) and verified by a VeriSign^®, the leading SSL Certificate authority.

A.11. Sensitive Questions

Grantees do not report on any data concerning sensitive topics.

A.12 Burden Estimate (Total Hours & Wages)

A.12A Estimated Annualized Burden Hours

The total annual burden is estimated to be 1,251.6 hours for grantees to collect, summarize, and report the data for the performance measures. Estimates were based on data collection timing from FY 2010 grantees.

Average burden hours for grantees or sub-awardees

All of the data will be reported by the grantees and their implementing sub-awardees twice a year. Beginning one year after receiving their grants, and every six months for the remainder of the 5-year grants, grantees will report performance measure data to OAH. Reporting of the performance measures coincides with the regular bi-annual reporting from OAH grantees- in January with their annual progress reports and in July with continuation applications. Data is reported twice a year to give OAH project officers managing these grants the ability to provide feedback related to implementation timely enough for grantees to make changes (many of the programs they are implementing are very short and if feedback was only provided yearly would not be as useful to grantees). Additionally, these grants are technically cooperative agreements, and therefore frequent government involvement is expected. Grantees already collect all of these data for their own enrollment/administrative purposes and are expected have their own systems in place to track the data (i.e. fidelity, attendance). Thus, the only additional burden to grantees for reporting the performance measures is the time it takes them to assemble the necessary data and enter it into the reporting forms. Respondents can either upload spreadsheets into the web-based system or enter the data directly into the system using a point and click method. The burden may be significantly less for grantees who manage their data using RTI-designed spreadsheets.

For grant programs Tier 1B, Tier 2B, and Tier 2C, grantees (or their designated reporter) will be the respondent. For grant programs Tier 1A and Tier 2A, a combination of the sub-awardee (or their designated reporter) and the awardee would be the respondents. Exhibit 4 provides the estimated number of respondents for each grant type. In computing burden for Tier 1A and Tier 2A grant programs, we used the number of sub-awardees to make our estimates, which represents the maximum level of burden.

Exhibit 4. Total Respondents for Each Funding Stream for FY 2015 Grants

TPP Program Funding Stream	# Grant Awards	# of Respondents
Tier 1A	8	322
Tier 1B	50	50
Tier 2A	2	323
Tier 2B	21	21
Tier 2C	3	3

Grantee- Level Measures

These are measures that are to be completed at the grantee or sub awardee level, which concern features of the project as a whole, rather than program model specific.

• Dissemination. Grantees or sub-awardees will report on measures of dissemination such as manuscripts published and papers presented. We estimate that this will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.

• Number of Partners. Grantees or sub-awardees will report on measures of number and retention of partners. We estimate that it will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.

• Training. Grantees or sub-awardees will report on the number of facilitators who receive initial and follow up training. We estimate that it will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.

• Linkages to Youth Friendly Health Care. Grantees or sub-awardees will report on the number of referrals to healthcare providers made by program staff on behalf of program participants. We estimate that it will take grantees or sub-awardees 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.

• Sustainability. This measure will be determined by pulling from responses to 1) number and type of partners (partner section); 2) amount of other funding available (cost section); and 3) Dissemination efforts (dissemination section). There is no separate reporting required from grantees or subawardees for OAH to create this measure.

• Cost. Grantees or sub-awardees will collect data regarding cost of implementing core and supplementary program services and monitoring their implementation as part of routine program management. We estimate that it will take respondents approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year for both the grantee-level and program-level measures.

Program-Level Measures

These are measures that summarize facets of the delivery of the program. Data will be reported separately by program model.

• Participants’ Reach. These data indicate the number of participants, by different background factors that the program is reaching (defined as participating in at least one program activity). We anticipate that it will take each respondent 2.8 hours per reporting period, or a total of 5.6 hours year to report these data.

• Dosage. Grantees or sub-awardees will collect attendance data on participants as part of their program management. The attendance data that are reported will be used to determine the percentage of core program components participants receive. We estimate that it will take respondents 1.7 hours to summarize and report these data per reporting period, for a total of 3.4 hours per year.

• Fidelity. Grantees or sub-awardees will collect several types of data related to fidelity as part of their ongoing administration of their programs. These include measures of adherence and quality, based on observations of 10% of the sessions implemented and a measure of adherence based on self-assessment forms completed by session facilitators; a measure of adherence based on the number of sessions implemented. OAH anticipates awarding 10 Tier 1B grantees within the maximum funding range; those grantees would be expected to serve at least 15,000 youth per year and would be required to observe 5% of sessions implemented. We estimate that it will take respondents approximately 1.7 hours to summarize and report these data per reporting period, for a total of 3.4 hours each year.

• Cost. Grantees or sub-awardees will collect data regarding cost of implementing core and supplementary program services and monitoring their implementation as part of routine program management. We estimate that it will take respondents approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.

Average burden hours to program participants

There is no burden on program participants.

Estimated annualized burden hours

Calculation of the total estimated annualized burden hours is shown in Exhibit 5. Data are reported twice a year; the average burden per response in the table is for each reporting period.

Exhibit 5. Estimated Annualized Burden Hours

Forms (If necessary)	Type of Respondent	Estimated Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours
Grantee- Level Measures
Dissemination	Grantee staff	84	2	.25	42
Number of partners	Grantee staff	84	2	.25	42
Number of facilitators trained	Grantee staff	84	2	.25	42
Health-care linkages	Grantee staff	84	2	.25	42
Program-Level Measures
Participant reach	Grantee/Sub-awardee staff	84	2	2.8	470.4
Dosage	Grantee/Sub-awardee staff	84	2	1.7	285.6
Fidelity	Grantee/Sub-awardee staff	84	2	1.7	285.6
Cost	Grantee/Sub-awardee staff	84	2	.25	42
Total					1,251.6

As calculated above, we estimate that it will take each of the 84 grantees or sub-awardees 1.25 hours per reporting period or 2.5 hours per year to report grantee-level measures of dissemination, partners, facilitators trained, linkages to youth-friendly health care, and sustainability. The total burden for reporting grantee-level measures for all respondents is 168 hours per year. It will take each of the 84 grantees or sub-awardees 6.45 hours per reporting period or 12.9 hours per year to report program-level measure of participant reach, dosage, fidelity, quality and cost per year for a total burden of 1083.6 hours. Thus, the total burden for all respondents is 1,251.6 hours.

A.12B Estimated Annualized Cost to Respondents

The estimated 1-year annualized cost to respondent is shown in Exhibit 6. Salaries of the grantee staff collecting data, entering data, and summarizing and reporting data will vary widely. We estimate an average hourly rate of $30.

Exhibit 6. Estimated 1-Year Annualized Cost to Respondents

Forms (If necessary)	Type of Respondent	Estimated Number of Respondents	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Grantee- Level Measures: Dissemination, Partner/Trainer Reach & Retention, Linkages,	Grantee/Sub-awardee staff	84	168	$30.00	$ $5,040
Program-Level Measures: Participant Reach, Dosage, Fidelity, Cost	Grantee/Sub-awardee staff	84	1083.6	$30.00	$ $32,508
Total					$37,548

A.13 Capital Costs (Maintenance of Capital Costs)

There are no capital costs associated with this study.

A.14 Cost to Federal Government

The cost to the federal government is $1,381,500 over 3.75 years (time remaining on contract) to complete the contract. This is the cost estimated by the contractor, RTI International, and includes the estimated cost of coordination with OAH, RTI IRB and OMB applications, revision of the data reporting system for the new cohort of grantees, on-going maintenance of the data reporting system, training and technical assistance to the grantees and OAH staff in the use of the data reporting system, and data analysis and reporting. These estimates are through September 2018, which is the end of the current contract with RTI. Annual cost to the government is $309,920 per year.

A.15 Program or Burden Changes

There is no change in burden requested, as this is a new information collection.

16. Tabulation of Data and Schedule

The FY 2015 cohort will first collect and report data in Spring 2016. The cohort will continue to collect and report data until Summer 2020 at the semi-annual required reporting periods (anticipated to be January 31 and July 30). RTI will then analyze the data and prepare a written report, summarizing findings. Data will be broken down by type of grantee (i.e., Tier 1A, Tier 1B, Tier 2A, Tier 2B, or Tier 2C) and by tier (Tier 1 and Tier 2).

A.17. Display of Expiration Date for OMB Approval

The expiration date for OMB will be displayed on all data collection instruments.

A.18. Exceptions to Certification Statement.

There are no exceptions to the certification statement.

Appendix A

Federal Register Notice to the Public

Appendix B

RTI Institutional Review Board Notice

Activity does not require IRB approval

Appendix C

Performance Measures

Appendix D

Program Observation (Quality) Form

Appendix E

Web-Shots of Data Entry Screens

1 These are new measures for the FY 2015 cohort of TPP grantees

2 This includes the 8 grantees and the expected 3 sub-awardees per grantee; the maximum number for any one question is anticipated to be 24.

3 This includes the 2 grantees and the expected 15 sub-awardees per grantee is anticipated to be 30.

20