Supporting Statement A - Justification for the Collection of Information for Local Evaluations as part of PREP PYP

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Justification for the Collection of Information for Local Evaluations as part of the Personal Responsibility Education Program (PREP): Promising Youth Programs (PYP)

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

February 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Calonie Gray

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request: This data collection will inform evaluation support provided to Personal Responsibility Education Program (PREP) grantees for their own local evaluations. The request includes two templates that ask grantees to summarize key information about their analysis plans and two templates to gather information about key lessons learned from conducting local evaluations. Responses to the analysis plan templates will support initial reviews of analysis plans to ensure they meet the standards for rigor expected for grantees’ final reports. The data collected from all instruments will help ACF understand the challenges grantees face when implementing and evaluating their programs to inform future grant programs and technical assistance. The data collected through the templates are not intended to generalize to a broader audience. We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

To improve the life course of adolescents and reduce the risks related to sexual activity, Congress authorized the Personal Responsibility Education Program (PREP) as part of the 2010 Affordable Care Act. It was reauthorized in 2015 for an additional two years of funding through the Medicare Access and CHIP Reauthorization ACT of 2015, which also mandates that the Secretary evaluate the programs and activities carried out with funds made available through PREP. This information collection request pertains to two programs funded through PREP—the Personal Responsibility Education Innovative Strategies Program (PREIS), and Tribal PREP (TPREP). PREIS supports the development of innovative strategies to prevent teen pregnancy for high-risk, vulnerable, and culturally underrepresented youth populations. Tribal PREP supports programming designed to reduce teen pregnancy and birth rates and the spread of STIs for American Indian/Alaska Native (AI/AN) youth.

In 2016, ACF contracted with Mathematica Policy Research to conduct the PREP: Promising Youth Programs (PYP) project, with the goal of supporting further development of the evidence base for teen pregnancy prevention programs. The PYP project has two broad objectives: (1) to provide evaluation support to grantees and their local evaluators and (2) to develop curricula for underserved youth to address sexual health and other PREP-related priorities. The data collected with the instruments in this information collection request (ICR) relates to the first objective. Specifically, it will give ACF and Mathematica information needed to provide technical assistance to PREIS and TPREP grantees on the analysis plans for their local evaluations and gather information about grantees’ experiences working with vulnerable, hard-to-reach populations to inform future ACF teen pregnancy prevention programming.

A2. Purpose

Purpose and Use

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (OMB #0970-0531):

• Delivery of targeted assistance related to program implementation or the development or refinement of program and grantee processes.

• Planning for provision of programmatic and evaluation-related technical assistance (T/TA).

Overview of Information Collection

The collection proposed under this ICR is for grantee organizations in the PREIS and Tribal PREP programs (13 and 8 grantees, respectively). The information being collected using these instruments is for program and evaluation monitoring and technical assistance only. There is no independent study being conducted using the information collected. ¹

Analysis plan templates: In accordance with grant requirements, grantees must complete an analysis plan to help evaluators produce rigorous evaluation reports.  The templates are a means of ensuring that analyses are specified a priori, to demonstrate a commitment to objectivity and to a systematic, scientific approach. Pre-specifying an analytic approach can prevent concerns about data mining. Grantees will complete these plans as they wrap up implementation and data collection activities and begin thinking about producing a final report. The analysis plan templates will prepare grantees for drafting a final report by asking grantees to document their intended analytic methods ahead of conducting the analyses and writing the report. Once finalized, the information collected will serve as the plan for conducting the analysis that supports the final evaluation report.

Lessons learned and promising practices templates: In accordance with grant requirements, grantees must complete the lessons learned and promising practices template in order to document key lessons learned from implementing their programs and conducting local evaluations.  Many of the grantees are working with vulnerable, hard-to-reach populations, where there is minimal research in the teen pregnancy field on how to serve these youth. A goal of both the PREIS and Tribal PREP grants is to help inform how teen pregnancy prevention programs can best serve these vulnerable populations. Information gathered from these templates will be shared with current grantees and used to inform ACF about how best to assist future grantees in improving their programming and evaluation. To fully understand the lessons learned presented by grantees in the template, ACF and the contractor will hold one follow-up conference call with each grantee to clarify responses to the lessons learned and promising practices template.

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Analysis plan templates	Instrument 1: PREIS Impact Analysis Plan Template Instrument 2: Tribal PREP Descriptive Analysis Plan Template	Respondents: 13 PREIS grantees and 8 Tribal PREP grantees Content: Requests information on methods grantees will use to analyze outcomes and implementation data for their local evaluations Purpose: To ensure that analyses are methodologically sound and specified a priori, to demonstrate a commitment to objectivity and to a systematic, scientific approach.	Mode: Electronic written document Duration: 14 hours (this includes time for initial completion and 2 revisions)
Lessons learned and promising practices templates	Instrument 3: PREIS Lessons Learned and Promising Practices Template Instrument 4: Tribal PREP Lessons Learned and Promising Practices Template	Respondents: 13 PREIS grantees and 8 Tribal PREP grantees Content: Requests information on barriers and challenges encountered during grant period, strategies to address them, and lessons learned from experience Purpose: To inform ACF about how best to assist future grantees in improving their programming and evaluation	Mode: Electronic written document and conference call Duration: 5 hours (this includes time to complete the template and participate in a follow-up discussion)

Other Data Sources and Uses of Information

The project received OMB approval in September 2016 to collect grantees’ evaluation design plans under the generic clearance for Formative Data Collections (OMB #0970-0356). The project also received OMB approval in 2017 (OMB #0970-0504; November 30, 2020) for collection of information related to grantees’ ongoing data collection and a project abstract, along with topic guides related to the second objective of the PYP project-- developing curricula for underserved youth to address sexual health and other PREP-related priorities. Additional ICR(s) will be submitted for this project in late 2020 for final evaluation report templates based on the analyses outlined in the analysis plans for both PREIS and Tribal PREP grantees, as well as for approval to use data collection instruments for the curriculum development tasks in conjunction with the second objective of the PYP project.

A3. Use of Information Technology to Reduce Burden

ACF and its contractors will employ information technology as appropriate and whenever possible to reduce the burden of respondents who agree to participate. This data collection effort will make use of electronic templates for completion and submission.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

This ICR is one of several related to carrying out the current round of grants funded by the Personal Responsibility Education Program, but this is the only one that will address analysis plans and lessons learned for the grantees in question. See section A2 for additional information about related ICRs.

Grantees may be able to use other documents (such as their evaluation design plans) to pull information to complete their analysis plan templates. This will minimize their burden in filling out the analysis plan template as they can rely somewhat on previously written information. However, none of the existing documentation on this project has collected all of the information collected in the analysis plan template, particularly the detailed description of how they plan to analyze the data for their evaluations (which is important for them to pre-specify for the field to signal scientific integrity). No other documents previously have asked grantees to provide information on promising practices or lessons learned. Therefore, no unnecessary information is being requested of program staff in the PREIS and Tribal PREP grants.

A5. Impact on Small Businesses

The potential exists for data collection activities to affect small entities, that is the grantees themselves or organizations associated with the grantees. Grantee organizations may be small businesses. PREP grantees may hire local evaluators that are part of a small business. Grantees may task the local evaluator with the collection of some or all of the data requested. Proposed data collection efforts are designed to minimize the burden on all organizations involved, including small businesses and entities, by collecting only critical information through the use of standardized templates.

A6. Consequences of Less Frequent Collection

We expect that grantees will complete the analysis plan template once and then make up to two revisions resulting from the provision of technical assistance during the grant period. Collecting this data less frequently would mean that the government and contractor could not determine whether the plans will meet the grantee’s research objectives as well as the standards of rigor set by ACF. In addition, if grantees did not submit revisions to their analysis plans, we would not have a complete and finalized plan for each evaluation, which would hamper our ability to determine potential flaws in the analyses as they evolve over time.

Grantees will complete the lessons learned and promising practices template one time, followed by one follow-up conference call to provide clarifications to the written responses in the template. If grantees did not complete the template or participate in the conference call, ACF would not have a complete understanding of their perspective of the successes and challenges they encountered, which would hamper ACF’s ability to understand their experience and identify potential strategies for future grantees.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. A subsequent notice, updated with more specific information, was published on June 18, 2019, Volume 84, Number 117, page 28307, and provided a thirty-day period for public comment.

Consultation with Experts Outside of the Study

When preparing the templates, ACF consulted staff from Mathematica Policy Research:

• Jean Knab, PYP Project Director

• Christine Ross, PYP Principal Investigator

• Juliette Henke, PYP Deputy Project Director

• Kristin Hallgren, Senior Researcher

• Julieta Lugo-Gil, Senior Researcher

A9. Tokens of Appreciation

No tokens of appreciation for respondents are proposed for this information collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

No personally identifiable information will be collected. We will not collect information regarding the individual respondent for any template.

Assurances of Privacy

As specified in the contract, Mathematica (the Contractor) shall protect grantee privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. Grantees will be informed about the plans on how the data will be used, and that their information will be kept private to the extent permitted by law.

Data Security and Monitoring

The Contractor shall ensure that all of its employees, including employees of all subcontractors, who perform work under this contract are trained on data privacy issues and comply with the above requirements. All Mathematica staff are required to sign the Mathematica Staff Confidentiality Agreement and participate in annual security awareness training.

The Contractor will use a data access plan consistent with OMB Policy and the Foundations for Evidence-based Policy Making Act of 2018. The data collected through the templates will not be shared outside of the federal and contractor staff directly involved with the grantees.

A11. Sensitive Information

There are no sensitive questions in this data collection.

A12. Burden

Explanation of Burden Estimates

Table A12.1 provides the estimated annual reporting burden calculations for the four instruments included in this request. The total burden is estimated to be 420 hours. Assumptions by instrument follow.

• PREIS Impact analysis plan template (Instrument 1). Thirteen grantees will complete the impact analysis plan template (grantees with impact evaluations), with two revisions to the initial draft. Respondents will first complete the full template, which we estimate will take 8 hours. We anticipate that the first revision may take respondents up to four hours and the second revision may take up to two hours. Averaged over the three responses, the average burden per response is five hours. The estimated total burden for this effort is 195 hours.

• Tribal PREP Descriptive analysis plan template (Instrument 2). Eight grantees will complete the descriptive analysis plan template (grantees with descriptive evaluations), with two revisions to the initial draft. Respondents will first complete the full template, which we estimate will take 8 hours. We anticipate that the first revision may take respondents up to four hours and the second revision may take up to two hours. Averaged over the three responses, the average burden per response is five hours. The estimated total burden for this effort is 120 hours.

• PREIS Lessons learned and promising practices template (Instrument 3). All 13 PREIS grantees will complete the lessons learned and promising practices template and participate in one conference call. On average, it will take 4 hours to complete this template and one hour to participate in the conference call. The estimated total burden for this effort is 65 hours.

• Tribal PREP Lessons learned and promising practices template (Instrument 4). All 8 Tribal PREP grantees will complete the lessons learned and promising practices template and participate in one conference call. On average, it will take 4 hours to complete this template and one hour to participate in the conference call. The estimated total burden for this effort is 40 hours.

Estimated Annualized Cost to Respondents

The estimated cost to respondents is $13,192.20. For cost calculations for the labor associated with completing the analysis plans, we estimate the average hourly wage for program directors and managers to be the average hourly wage for “Social and Community Services Manager” ($31.41), taken from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics, 2018.²

Table A12.1

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Impact analysis plan template	13	3	5	195	$31.41	$6,124.95
Descriptive analysis plan template	8	3	5	120	$31.41	$3,769.20
Lessons learned and promising practices template - PREIS	13	1	5	65	$31.41	$2,041.65
Lessons learned and promising practices template – Tribal PREP	8	1	5	40	$31.41	$1,256.40
Total	42			420		$13,192.20

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total cost for the data collection activities under this current request will be $259,396. We estimate the cost for each study activity in the table below. Estimated costs include contractor staff labor hours; operational expenses including equipment, overhead, printing, staff support, and travel; and any other expenses which would not have been incurred without this collection of information.

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$28,152
Analysis	$231,244
Total costs	$259,396

A15. Reasons for changes in burden

This is a new information collection request under the Generic Clearance for Formative Data Collections (OMB #0970-0531).

A16. Timeline

Activity	Date
Receive OMB approval	Winter 2020
Grantees will receive initial drafts of templates	Spring 2020
First draft analysis plan and lessons learned and promising practices templates due	Late Spring/Early Summer 2020
Updated analysis plan due	Fall 2020
Conference calls following up on lessons learned and promising practices responses	Fall 2020

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

• Instruments:

  • Instrument 1: PREIS Impact Analysis Plan Template

  • Instrument 2: Tribal PREP Analysis Plan Template

  • Instrument 3: PREIS Lessons Learned and Promising Practices Template

  • Instrument 4: Tribal PREP Lessons Learned and Promising Practices Template

• Appendices:

  • Appendix A: Section 513 of Social Security Act

  • Appendix B: ACF Personal Responsibility Education Program (PREP): Promising Youth Programs (PYP)— PREIS Impact Analysis Plan Instructions

  • Appendix C: ACF Personal Responsibility Education Program (PREP): Promising Youth Programs (PYP)—Tribal PREP Analysis Plan Instructions

1 The data to be collected in this ICR, particularly the analysis plan templates, will build on the data collected under OMB #0970-0504. Under OMB #0970-0504, grantees provide occasional updates on the status of their evaluation enrollment and data collection efforts. These updates help monitor the state of evaluations and inform technical assistance efforts. The analysis plan templates will collect detailed information on how grantees plan to analyze the data they’ve collected from the sample they enrolled. ACF and Mathematica will review the analysis plans informed by the data provided through those updates to ensure their methods are scientifically sound.

2 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Social and Community Service Managers, on the Internet at https://www.bls.gov/ooh/management/social-and-community-service-managers.htm (visited Jul 22, 2019).

10