0990_0438_TPP_Performance Measures_Renewal_Part_A_Supporting_Statement_clean2018v4
0990_0438_TPP_Performance Measures_Renewal_Part_A_Supporting_Statement_clean2018v4.docx
Office of Adolescent Health Teen Pregnancy Prevention, FY 2015-2020 Performance Measure Collection
OMB: 0990-0438
Supporting Justification for the OMB Revision of Clearance of the
“Office of Adolescent Health Teen Pregnancy Prevention, FY 2015-2020 Performance Measure Collection”
Part A: Justification for the Collection of the Data
July 2018
Submitted by:
Office of Adolescent Health
U.S. Department of Health and Human Services
1101 Wootton Parkway, Suite 700
Rockville, MD 20852
Project Officer: Tara Rice
Background
The Office of Adolescent Health (OAH) is requesting revision of approval for the collection of performance measures for the Teen Pregnancy Prevention (TPP) Program grantees (OMB # 0990-0438). A 3 year clearance period is requested.
The TPP Program was initiated in FY 2010 as one of six major evidence-based policy initiatives across the Federal government. OAH supports two types of grants through the TPP program (1) projects that replicate evidence-based TPP program models that have been shown to be effective through rigorous evaluation, referred to as Tier 1 and (2) research and demonstration projects that develop and test additional models and innovative strategies to prevent teen pregnancy, referred to as Tier 2. Funding for the TPP Program is authorized under the Continuing Resolution for FY 2018 (Public Law 115-124). The Act provides $100 million in FY 2018 for making competitive awards to public and private entities to fund programs that reduce teen pregnancy. In FY 2018, OAH anticipates funding up to 84 cooperative agreements. Detailed information, including the anticipated number of awards in each category may be found in Exhibit 1.
Performance measures have consistently been a requirement of the TPP grant program since its inception. The measures that are included in this information collection request previously have received OMB clearance (0990-0392 and 0990-0438) and have been used successfully over the past 6 years with the initial 2 cohorts of TPP grantees.
This Supporting Statement requests revision of approval for collection of performance measures for the second cohort of TPP grantees. The number of estimated respondents have been adjusted to reflect changes in the number of grantees funded.
The performance measures data collection will fulfill several important functions. First, OAH expects its grantees to use the measures to make continuous quality improvement in their program implementation and inform their partners and stakeholders about implementation and sustainability progress. Second, performance measures provide OAH with metrics for monitoring TPP grantees so that project officers can provide technical assistance when needed. Finally, OAH uses the measures to report to OASH, OS, our budget office, and Congress on the grantees’ progress.
Exhibit 1: Estimated Number of TPP Projects to be funded in FY 2018
New TPP Project Funding Opportunity in FY 2015 |
Estimated Total Annual Funding |
Estimated # of Awards |
Estimated Annual Award Amount |
Rigorous Evaluation |
Capacity Building to Support Replication of Evidence-Based TPP Programs (Tier 1A) |
$5 million |
8 |
$400,000 - $750,000 |
No |
Replicating Evidence-Based TPP Programs to Scale in Communities with the Greatest Need (Tier 1B) |
$60 million |
50 |
$500,000 - $2,000,000 |
No |
Supporting and Enabling Early Innovation to Advance Adolescent Health and Prevent Teen Pregnancy (Tier 2A) |
$3 million |
2 |
$1,000,000 -$1,500,000 |
No |
Rigorous Evaluation of New or Innovative Approaches to Prevent Teen Pregnancy (Tier 2B) |
$18 million |
21 |
$400,000 - $1,000,000 |
Yes |
Effectiveness of TPP Programs Designed Specifically for Young Males (Tier 2C) |
$2 million |
3 |
$600,000 - $1,000,000 |
Yes |
A1. Need and Legal Basis
The performance measure collection is critical to OAH because it provides the agency with data to both effectively manage the TPP program, and to comply with accountability and federal performance requirements for the 1993 Government Performance and Results Act (P.L. 103-62). Moreover, collecting and reporting on data for performance measures are funding requirements for the grants, as stated in the funding opportunity announcement.
The performance measures to be reported to OAH are summarized in Exhibit 2. The specific questions that grantees will answer to address these measures are found in Appendix C.
Exhibit 2: Performance Measures for the FY 2015-2020 Cohort
Performance Measure Constructs |
Data source |
|
|
Grantee-Level Measures |
|
Dissemination |
Grantee/Sub-awardee Administrative Records |
|
|
|
|
Number of Partners |
|
|
Grantee/Sub-awardee Administrative Records |
|
|
Training |
Grantee/Sub-awardee Administrative Records |
|
|
|
|
Linkages to Youth Friendly Health Care |
|
|
Grantee/Sub-awardee Administrative Records |
Sustainability1 |
|
|
Grantee/Sub-awardee Administrative Records |
|
|
Program-Level Measures |
|
Participant Reach |
|
|
Grantee/Sub-awardee Administrative Records |
Dosage |
Grantee/Sub-awardee Attendance Records |
|
|
|
|
Fidelity |
|
|
Fidelity Monitoring Logs |
|
|
|
Observation Forms |
|
|
Cost |
|
|
Grantee/Sub-awardee Administrative Records |
|
A.2 Information Users
The data collection activities will provide information to OAH leadership and program officers to help them to more effectively manage the TPP program. We anticipate that the data will be made available to Congress by the Office of Management and Budget (reach, dosage, training and partners), and the public at large (all measures) to assess program performance. Use of these data is vital for ensuring on-going improvement of the TPP program and through dissemination efforts, broader understanding and support of programs designed to prevent teen pregnancy.
A.3. Use of Information Technology and Burden Reduction
Grantees will enter performance measure data into a multi-use, Web-based reporting system (similar to the system used by the previous grant cohort), either by using point and click entry or by uploading spreadsheets using a template. The Web-based system can reduce burden for respondents by programming in skip patterns, so that grantees only have to look at questions or uploading data that are relevant for them. Programming automatically performs necessary calculations for respondents, and will validate responses. For point and click entry, a branching mode of presentation allows respondents to go directly to the sections they need, without having to go through the system in a linear progression. The system also automatically produces a data set of measures across all grantees using relevant filters (e.g., for Tier 2B grantees only), which saves time on preparation of the data for analysis. Data are also available for grantees to export into Excel files to custom design reports. Screenshots of the current web-based reporting system are included in Appendix E.
A.4. Efforts to Identify Duplication and Use of Similar Information
The OAH performance measures data collection is the only data collection that will provide information on the performance of the TPP program specifically. The data collection will make use of existing data to the extent possible. Most of the performance measures would be collected by grantees as part of their routine administrative records (e.g., numbers of publications, numbers of participants, and attendance at program sessions). Please see Exhibit 2 for data sources for each performance measure and see Appendix C for the complete list of measures.
The current request is a revision to renewal with updated burden calculation.
A.5. Impact on Small Businesses
No small businesses will be involved in the collection of data in this study.
A.6. Consequences of Not Collecting the Information/Collecting Less Frequently
GPRA requires that government agencies report on their performance measures annually. Therefore, it is essential that grantees report on these performance measures at least annually to OAH. Biannual reporting allows OAH to provide technical assistance to the TPP grantees, and for grantees to use data for continuous quality improvement. In addition, collection and reporting of performance measure data is a requirement of all TPP grantees as stated in the Funding Opportunity Announcement.
A.7. Special Circumstances
There are no special circumstances that occur when collecting this information.
A.8. Federal Register Comments and Persons Consulted Outside the Office of Adolescent Health
A 60-day notice was published in the Federal Register on April 5, 2018, in Volume 83, Number 66, pages 14643-14644, and provided a 60-day period for public comments (Appendix A). Two public comments were received. The text of the letters is included below, and the signed letters are included in Appendix F):
Letter from the National Association of County and City Health Officials NACCHO, dated June 4 2018
“Dear Mr. Azar:
The National Association of County and City Health Officials (NACCHO) is pleased to submit the following response to Information Collection Request 83 FR 14643 concerning data collection for the teen pregnancy prevention program (TPPP).
As the national voice for the nation’s nearly 3,000 local health departments, NACCHO is a leader, partner and catalyst for local health departments to ensure that people are able to make choices that make them healthier. Among their various functions, local health departments protect, promote, and improve the health of women, children, adolescents and families.
TPPP was established in 2010 to support community-driven, evidence-based or informed, medically accurate, and age-appropriate approaches to preventing pregnancy among adolescents, involving parents, educators, researchers, and providers. In the first five years of TPPP alone, more than 7,000 professionals were trained, 3,000 community-based partnerships were developed, eight new innovative programs were identified as contributing to positive health behavior change, and over half a million young people were served.
As the future of the program is discussed by the Department of Health and Human Services (HHS), it is important to have data to be able to assess the outcomes that TPPP has produced. The latest performance measures proposed by HHS are similar to data on the program that has previously been collected. NACCHO is supportive of the new proposed performance measures because they will allow comparison to previous evaluation.
However, NACCHO suggests that additional information is required to adequately assess the impact of TPPP. Key points such as demographic information of program participants and whether appropriate linkages to health care providers were made are not included in the proposed data to be collected. It is important to know who TPPP is serving and whether the program is helping young people access clinical and other health and social services.
In regards to data to be collected and the associated time estimate, HHS should be sure to include adequate time in the estimate to ensure the data are complete and correct. Time is involved in not only the collection process, but also in verifying the accuracy of the data.
Thank you for the opportunity to provide input on data collection for TPPP. For more information, please contact Eli Briggs, Senior Government Affairs Director, at [email protected] or 202-507-4194.
Sincerely,
Lori Tremmel Freeman, MBA
Chief Executive Officer”
Letter from Big Cities Coalition, dated June 4, 2018
Dear Secretary Azar:
I am contacting you on behalf of the Big Cities Health Coalition (BCHC), which is comprised of health officials leading 30 of the nation’s largest metropolitan public health departments. Together, BCHC health departments serve more than 55 million – or one in six – Americans.
As you well know, the Teen Pregnancy Prevention Program (TPPP) was established in
2010 to support community-driven, evidence-based (or informed), medically accurate,
and age-appropriate approaches to preventing pregnancy among adolescents, involving
parents, educators, researchers, and providers.
In the first five years of TPPP alone, more than 7,000 professionals were trained, 3,000
community-based partnerships were developed, eight new innovative programs were
identified as contributing to positive health behavior change, and over half a million
young people were served. In this same time period, urban health departments have seen
great success in lowering teen pregnancy rates. Overall, teen birth rates in the U.S. have
dropped 8 percent since 2014, according to the CDC, and, again, cities have seen even
larger drops. Even so, the rate is still substantially higher than other industrialized nations.
As the future of the program is discussed at HHS, it is important to have accurate data to
assess the outcomes that TPPP has produced. The latest performance measures proposed
by the Department are similar to what has previously been collected, and BCHC is supportive of the new proposed performance measures because they will allow comparison to previous evaluation. However, we also believe additional information is needed to adequately assess the ongoing impact of TPPP. These are highlighted below.
Demographic information of program participants should be collected.
Information about whether appropriate linkages to health care providers were made
also need to be collected.
Both of these data points would help determine who TPPP is serving and whether
the program is helping young people access clinical and other health/social services.
Finally, in regards to data being collected and the associated time estimates, HHS
should be sure to include adequate time in the estimate to ensure that data are
complete and correct. Time is involved in not only the collection process, but also in
verifying the accuracy of the data being reported.
Thank you for the opportunity to provide input on data collection for TPPP, and please
do not hesitate to reach out to me at [email protected] or 202-783-3627.
Chrissie Juliano, MPP
Director, Big Cities Health Coalition
Both letters raised identical points about the data collection. In response to the points raised in the two letters, demographic information on program participants is already part of this data collection. Participant demographics in cohort 2 are being collected at the level of the section or group, rather than the individual level. OAH collected individual-level demographics from participants during cohort 1, and found that many organizations lacked the necessary data infrastructure to collect this data and report it to OAH in a timely fashion. As cohort 2 grantees serve a much larger number of participants annually than cohort 1 grantees did, OAH decided to collect participant demographics in aggregate (at the level of an individual class or group receiving a TPP intervention); this decision was made to reduce burden on grantee respondents, and has resulted in more timely data collection. Based on the challenges OAH faced in collecting individual-participant-level demographic data from TPP grantees during cohort 1, and the larger reach of the grants in cohort, OAH still believes that collecting demographic data at the aggregate level is the best choice for the purposes of this data collection.
Both letters commented on the need to collect information about healthcare linkages. The cohort 2 healthcare linkage item was not included in the 60-Day FRN draft. This item has been restored within this data collection and burden table. Both letters discuss the potential benefit of using individual demographics and health care referral information in combination to determine whether the TPP program is helping specific groups of young people access clinical and other health/social services. This point is duly noted; however, as described above, OAH has found greater success from the collection of aggregate demographics from its grantees. Getting complete data in a timely fashion is a greater priority for this data collection.
Both letters comment on timing of data collection, and the need to ensure accurate data. The due dates for data reporting will continue to be 1 month after the end of each reporting period, as stated within the Notice of Award.
In 2010 and 2011, OAH consulted with staff of RTI International, which was the contractor responsible for assisting OAH in developing the performance measures and performance measure reporting system, and a panel consisting of experts in the fields of performance measurement, teen pregnancy prevention, and evidence-based practice. In addition, OAH presented information on the performance measures to TPP grantees and their evaluators at two conferences, and solicited their input. OAH also consulted and received feedback from other Federal staff working in the area of teen pregnancy prevention from ASPE, ACF, and CDC. The bulk of the performance measures that are part of this clearance package have been successfully used during the past 7 years in the data collection for the “Office of Adolescent Health and Administration for Children, Youth and Families Teen Pregnancy Prevention Performance Measure Collection” as well as the 2015 version of this data request.
A list of individuals in the expert panel who provided input regarding the process evaluation is found in Exhibit 3.
Exhibit 3. Persons Consulted Outside the Agency for Performance Measure Collection for FY 2010 Grantees |
||||||||||||||||||||
Expert Work Group |
||||||||||||||||||||
|
|
|||||||||||||||||||
|
|
|||||||||||||||||||
|
|
|||||||||||||||||||
|
|
|||||||||||||||||||
A.9. Payments to Respondents
There are no payments to staff of grantee organizations completing the performance measure reporting form.
A.10. Assurance of Confidentiality
All data are reported de-identified and aggregated to the section (group or class) level. No personal identifiers will be used in the reporting of any data.
The Web-based reporting system has been designed to ensure the security of the data obtained. Electronic data are stored in a location within the RTI network that provides the appropriate level of security based on the sensitivity or identifiability of the data. No personal identifiers will be used in the reporting of any data.
Individual users designated by the grantees will be assigned user names and passwords that will grant them access to the project website. There, users will have the opportunity to provide data that will be stored in a secure Microsoft SQL Server database utilizing a relational table structure, facilitating expedient data retrieval and analysis. The database server, located at RTI, will be accessible only to the statisticians and analysts assigned to this project. Electronic communications will occur via a secure Internet connection. All transmissions will be encrypted with 128-bit encryption through secure socket layers (SSL) and verified by a VeriSign®, the leading SSL Certificate authority.
A.11. Sensitive Questions
Grantees do not report on any data concerning sensitive topics.
A.12 Burden Estimate (Total Hours & Wages)
A.12A Estimated Annualized Burden Hours
The total annual burden is estimated to be 1252 hours for 84 grantees to collect, summarize, and report the data for the performance measures. Estimates were based on data collection timing from the previous cohorts of TPP grantees.
Average burden hours for grantees
Most of the data will be reported by the grantees or their designated reporters twice a year (with the exception of cost data, reported annually, and one sustainability item reported once at the end of the grant period). Grantees will collect most of these data for their own administrative purposes and are expected have their own systems in place to track the data. Thus, the only additional burden to grantees for reporting the performance measures is the time it takes them to assemble the necessary data and enter it into the reporting forms. Respondents can enter the data directly into the performance measures database system using a point and click method.
For Tier 1 and Tier 2 grant programs, grantees (or their designated reporter, such as an independent evaluator) will be the respondent. Exhibit 4 provides the estimated number of respondents for each grant type.
Exhibit 4. Total Respondents for Each Funding Stream for FY 2015 Grants
TPP Program Funding Stream |
# Grant Awards |
# of Respondents |
Tier 1A |
8 |
8 |
Tier 1B |
50 |
50 |
Tier 2A |
2 |
2 |
Tier 2B |
21 |
21 |
Tier 2C |
3 |
3 |
Grantee- Level Measures
These are measures that are to be completed at the grantee or sub-awardee level, which concern features of the project as a whole, rather than program model specific.
Dissemination. Grantees will report on measures of dissemination such as manuscripts published and papers presented. We estimate that this will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.
Number of Partners and Sustainability. Grantees will report on measures of number and retention of partners. We estimate that it will take each respondent approximately 0.25 hour to summarize and report these data each reporting period, for a total of 0.5 hours per year.
For sustainability, grantees will report about the activities and resources devoted to sustaining their programs after the grant has been completed. During most reporting periods, there will not be a specific unique measure for this item, and sustainability metrics will be determined by pulling from responses to 1) number and type of partners (partner section); 2) amount of other funding available (cost section); and 3) Dissemination efforts (dissemination section).
However, a single item, asked once at the final data collection for the grant cycle, grantees will be asked to indicate the number of partners with firm plans to continue the programming after funding ends.
Training. Grantees will report on the number of facilitators who receive initial and follow up training. We estimate that it will take each respondent approximately 0.25 hour (15 minutes) to summarize and report these data each reporting period, for a total of 0.5 hours per year.
Linkages to Youth Friendly Health Care. Grantees or sub-awardees will report on the number of referrals to healthcare providers made by program staff on behalf of program participants. We estimate that it will take grantees or sub-awardees 0.25 hour (15 minutes) to summarize and report these data each reporting period, for a total of 0.5 hours per year.
Program-Level Measures
These are measures that summarize facets of the delivery of the program. Data will be reported in aggregate for each section (group or class facilitated).
Participants’ Reach. These data indicate the number of participants, by different background factors that the program is reaching (defined as participating in at least one program activity). We anticipate that it will take each respondent 2.8 hours per reporting period, or a total of 5.6 hours year to report these data.
Dosage. Grantees will collect attendance data on participants as part of their program management. The attendance data that are reported will be used to determine the percentage of core program components participants receive. We estimate that it will take respondents 1.7 hours (1 hour 42 minutes) to summarize and report these data per reporting period, for a total of 3.4 hours per year.
Fidelity and Quality. Grantees will collect several types of data related to fidelity as part of their ongoing administration of their programs. These include measures of adherence and quality, based on observations of 5%-10% of the sessions implemented and a measure of adherence based on self-assessment forms completed by session facilitators; a measure of adherence based on the number of sessions implemented. We estimate that it will take respondents approximately 1.7 hours (1 hour 42 minutes) to summarize and report these data per reporting period, for a total of 3.4 hours each year.
Cost: Grantees will collect data regarding cost of implementing core TPP program services and data on additional sources of funding for the projects. We estimate that it will take respondents approximately 0.5 hour (30 minutes) to summarize and report these data once per year.
Average burden hours to program participants
There is no burden on program participants.
Estimated annualized burden hours
Calculation of the total estimated annualized burden hours is shown in Exhibit 5. Data are generally reported twice a year (except for cost and one of the partners/sustainability items as noted on page 14); the average burden per response in the table is for each reporting period.
Exhibit 5. Estimated Annualized Burden Hours
Respondent |
Forms
|
Estimated Number of Respondents |
Number of Responses per Respondent |
Average Burden Hours per Response |
Total Burden Hours |
TPP Grant Recipient |
Dissemination |
84 |
2 |
15/60 |
42 |
TPP Grant Recipient |
Number of partners & Sustainability |
84 |
2 |
12/60 |
42 |
TPP Grant Recipient |
Number of facilitators trained |
84 |
2 |
15/60 |
42 |
TPP Grant Recipient |
Health-care linkages |
84 |
2 |
15/60 |
42 |
TPP Grant Recipient |
Participant reach |
84 |
2 |
168/60 |
470 |
TPP Grant Recipient |
Dosage |
84 |
2 |
102/60 |
286 |
TPP Grant Recipient |
Fidelity |
84 |
2 |
102/60 |
286 |
TPP Grant Recipient |
Cost |
84 |
1 |
15/60 |
42 |
Total |
|
|
|
|
1,252 |
As calculated above (Exhibit 5), we estimate that it will take each of the 84 grantees 1 hour per reporting period or 2 hours per year to report grantee-level measures: dissemination, partners, facilitators trained, and health care linkages. The total burden for reporting grantee-level measures for all respondents is 189 hours per year. It will take each of the 84 grantees 5.6 hours per reporting period or 12.2 hours per year to report program-level measure of participant reach, dosage, fidelity, quality, and cost for a total burden of 1,045 hours. Thus, the total annualized estimated burden hours would be 1252 hours.
A.12B Estimated Annualized Cost to Respondents
The estimated 1-year annualized cost to respondent is shown in Exhibit 6. Salaries of the grantee staff collecting data, entering data, and summarizing and reporting data will vary widely. We estimate an average hourly rate of $30.
Exhibit 6. Estimated 1-Year Annualized Cost to Respondents
Forms
|
Type of Respondent |
Estimated Number of Respondents |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Grantee- Level Measures: Dissemination, Partners and Sustainability, Training, Health Linkages, Sustainability |
Grantee/Sub-awardee program staff |
84 |
189 |
$30.00 |
$ 5670 |
|
|
|
|
|
|
Program-Level Measures: Participant Reach, Dosage, Fidelity, Cost |
Grantee/Sub-awardee program staff |
84 |
1,045 |
$30.00 |
$31350 |
|
|
|
|
|
|
Total |
|
|
|
|
$36630 |
A.13 Capital Costs (Maintenance of Capital Costs)
There are no capital costs associated with this study.
A.14 Cost to Federal Government
The cost to the federal government is estimated at $1,381,500 over 3 years. This cost was estimated by using the current contract and includes the estimated cost of coordination with OAH, the contractor’s IRB and OMB applications, revision of the data reporting system for the new cohort of grantees, on-going maintenance of the data reporting system, training and technical assistance to the grantees and OAH staff in the use of the data reporting system, and data analysis and reporting. Annual cost to the government is $309,920 per year.
A.15 Program or Burden Changes
The primary change within this data collection is to extend the period of collection.
16. Tabulation of Data and Schedule
The cohort 2 of TPP grantees will continue to collect and report data throughout the grant cycle, which could be through the summer 2020 at the prescribed semi-annual required reporting periods (February 1 and August 1). OAH or its designated contractor will then analyze the data and prepare a written report, summarizing findings on an annual basis. Data will be summarized overall and/or broken down by tier (Tier 1 and Tier 2).
A.17. Display of Expiration Date for OMB Approval
The expiration date for OMB will be displayed on all data collection instruments.
A.18. Exceptions to Certification Statement.
There are no exceptions to the certification statement.
Appendix A
Federal Register Notice to the Public
Appendix B
RTI Institutional Review Board Notice
Activity does not require IRB approval
Appendix C
List of Performance Measures
Appendix D
Program Observation (Quality) Form
Appendix E
Web-Shots of Data Entry Screens
Appendix F
Public Comments Signed Letter #1
Appendix G
Public Comments Signed Letter #2
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for |
| Author | wallace |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |
© 2024 OMB.report | Privacy Policy