Generics SSA - CIS Proof of Concept

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Sexual Risk Avoidance Education National Evaluation

Program Components Impact Study:

Proof of Concept Pilot Phase

Formative Data Collections for Program Support

0970–0531

Supporting Statement

Part A

November 2021

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Jessica Johnson

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request:

As a pilot phase for the Sexual Risk Avoidance Education National Evaluation (SRAENE) Program Components Impact Study, these proof-of-concept pilot studies will collect information to plan for, strengthen, and determine feasibility of conducting future summative evaluations investigating how promising program delivery strategies for facilitators of sexual risk avoidance education (SRAE) programs impact facilitator and youth outcomes. For this phase of the project, we are seeking clearance to train facilitators on two program delivery strategies and to conduct rapid-cycle learning activities informed by facilitator surveys, facilitator interviews, and youth participant focus groups to develop and refine the two trainings and strategies for potential future summative evaluation. We do not intend the information to be used as the principal basis for public policy decisions.

• Time Sensitivity: ACF will use the data from this generic information collection to refine the program delivery strategies and assess the feasibility of beginning summative evaluations of either strategy in spring 2022. To remain on track for the study timeline, the proof-of-concept pilot studies must begin in early 2022.

A1. Necessity for Collection

In February 2018, as part of the federal government’s efforts to support youth in making healthy decisions about their relationships and behaviors, Congress reauthorized Title V, Section 510 of the Social Security Act to fund the Sexual Risk Avoidance Education (SRAE) grant program. SRAE grants fund programs that teach adolescents to refrain from sexual activity. The Family and Youth Services Bureau (FYSB), Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services (HHS), administers the program. SRAE programs also provide education on personal responsibility, self-regulation, goal setting, healthy relationships, a focus on the future, and preventing drug and alcohol use. The reauthorization also included a requirement to evaluate the SRAE grant program. ACF awarded a contract for the SRAE National Evaluation (SRAENE) in 2018.

This request pertains to one of the SRAENE activities – the Program Components Impact Study. The Program Components Impact Study will refine and test improvements to one or more components of programs to ultimately improve youth outcomes. Program components can include any part of a program, including curriculum content, supplementary activities, delivery, facilitation, setting, and dosage. This proposed generic information collection (GenIC) requests approval to conduct two proof-of-concept studies, each piloting a different facilitation strategy (described in A2) designed to improve the delivery of SRAE programs to youth. The proof-of-concept studies are needed to refine and strengthen training and implementation of the facilitation techniques and to determine feasibility ahead of potential future full scale summative evaluations. If the pilot suggests that the techniques can be replicated at the scale needed for summative evaluations, we will submit a fullInformation Collection Request (ICR) under the Program Components Impact Study to determine if the strategies improve youth outcomes. There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Planning for provision of programmatic or evaluation-related training or technical assistance (T/TA).

• Use of rapid-cycle testing activities to strengthen programs in preparation for summative evaluation.

Through various activities conducted during the course of implementing the SRAENE contract, ACF and its contractor, Mathematica, have received feedback from SRAE grantees and stakeholders that improving the facilitation of SRAE programs is a necessary first step for overall program improvement. With this request, ACF proposes to implement two facilitator trainings with a select set of SRAE grantee programs funded by FYSB on two separate strategies that intend to improve the delivery of SRAE programs to youth. The study team will conduct rapid-cycle learning (RCL) activities to pilot and refine implementation of the two strategies. Below, we describe each strategy.

1. Co-Regulation Strategy. The first strategy requires that facilitators use six co-regulation strategies while delivering their existing curricular content. A prior formative evaluation completed as part of the Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Programs for Youth (SARHM, OMB Control Number 0970-0355) guided the development of these strategies and suggested they can be integrated into SRAE programs.¹ However, the initial formative evaluation involved just two sites that delivered relationship education programs to youth, meaning there could be differences in content and facilitation that differentiates this proposed formative work from the SARHM work. Data collected for this proof-of-concept study would expand upon the prior work by closely examining the training and other supports needed to implement this strategy with fidelity among grantees delivering SRAE programs.

2. Heritage Method Strategy. The second facilitation strategy, the Heritage Method, teaches facilitators to assess youths’ attitudes and beliefs on constructs associated with the delay of sexual initiation, which builds on prior research by Heritage Community Services (HCS) and its research team (Weed et al. 2008; Weed et al. 2005; Weed et al. 1992). The Heritage Method is used to prepare, assess, and supervise facilitators delivering the Heritage Keepers curriculum. Its developer, HCS, maintains that this method for preparing and supporting facilitators is a critical program component that has not been replicated by other curriculum developers or program trainers. Importantly, HCS views the Heritage Method as appropriate for use in many SRAE programs using different curricula, provided that the curriculum goals and content are similar. This proof-of-concept study would export the Heritage Method training to grantees implementing a different curriculum than Heritage Keepers to assess the ability to replicate the strategy with fidelity.

To date, a limited number of programs have used these strategies and their implementation has been limited to the programs that supported their development. The proof-of-concept studies will pilot the training on and use of these strategies among a new set of grantee programs.

This proposed GenIC includes facilitator surveys, facilitator interviews, and youth participant focus groups. Through two RCLs, the study team will work with a small number of SRAE programs to implement each strategy, collect data, and collaborate with program staff to interpret findings and identify ways to refine the implementation of each strategy.

The lessons learned from the RCLs will inform the feasibility and design of two potential future summative studies to measure whether each strategy can succeed in improving facilitation and youth outcomes. ACF and their grantees will be able use the information collected through this request to revise and strengthen the facilitation strategies in advance of these two possible future summative evaluations. One full ICR for these two summative evaluations would be developed and submitted to OMB.

The information collected is meant to contribute to the body of knowledge on ACF programs. However, federal decision-makers will not use the collected information as the principal basis for creating policy or making policy decisions. Moreover, the information is not expected to meet the threshold of influential or highly influential scientific information.

Study Objectives

Both proof-of-concept studies have three common objectives:

• To implement and assess a facilitator training on each of the two facilitation strategies

• To identify and test any ongoing supports needed to strengthen the implementation of the two strategies

• To assess the feasibility of replicating the training and use of ongoing support with fidelity on a larger scale to conduct summative evaluations

Study Design

To conduct the proof-of-concept pilot studies, the team will recruit up to nine sites, with up to five sites specifically chosen for the co-regulation strategy study and a separate set of up to four sites specifically chosen for the Heritage Method study.

Sites will participate in facilitator training, use the strategies when implementing their SRAE program, and complete data collection activities. For more information about the sites to be recruited, see Supporting Statement B, Section B2.

Facilitators at recruited sites will receive training in one of the facilitation strategies. Facilitators will complete pre- and post-training facilitator surveys to learn about their experience with the training (Appendix A. Facilitator Survey Invitation Email; Instrument 1. Facilitator Pre-Training Survey; and Instrument 2. Facilitator Post-Training Survey). Following training, facilitators will implement the strategies into their SRAE programming. They will receive additional support to implement the strategies, as needed. During implementation, sites will participate in RCL to assess and refine use of the strategies. We expect that each site will complete two learning cycles. RCL activities are expected to take place over a three month period.

Each learning cycle will last approximately four weeks. During a cycle, facilitators will implement the strategy with a group of youth program participants. Toward the end of a cycle, study staff will conduct interviews with facilitators (Instrument 3. Facilitator Interview Guide) to discuss their perceptions of how the strategies are working, areas where they need more training or support, their ability and comfort in integrating the strategies into the curriculum, and their comfort and confidence in using the strategies in the classroom.

Near the end of the second learning cycle, youth program participants in both studies who have parental consent and assent (Appendix B. Parent Consent and Youth Assent Forms) will participate in focus groups (Instrument 4. Youth Focus Group Protocol). Focus groups will capture youth’s perceptions of the program, including their satisfaction with the program climate, topics covered in the class, and interactions with facilitators.

After the conclusion of each learning cycle, the study team will work with the sites to organize findings into themes and to reflect on the experience, identify additional lessons, and identify opportunities for additional improvement. Together, the study team and site staff will refine the strategy and plan for the next iterative learning cycle where the same process is repeated.

Universe of Data Collection Efforts

The current request includes three main data collection efforts (Table A.2).

Table A.2. Universe of data collection activities

Data collection activity	Instruments	Respondent, content, purpose of collection	Mode and duration
Facilitator survey	Instrument 1. Facilitator Pre-Training survey Instrument 2. Facilitator Post-Training Survey	Respondents: Professional facilitators Content: Facilitators’ understanding of the strategy tested and perceived self-efficacy for implementing strategy Purpose: To capture facilitator understanding of the strategy before and after training and to explore perceptions of and experiences with training	Mode: Web Duration: 10 minutes per survey
Facilitator interviews	Instrument 3. Facilitator Interview Guide	Respondents: Professional facilitators Content: Feedback on training, guidance, and materials; use of strategy; youth responsiveness; effectiveness of strategy; and suggestions for improvement Purpose: To determine how the strategy is being used and facilitators’ perceptions and experiences with using the strategy	Mode: In-person or remote (videoconference or telephone) Duration: Up to 45 minutes per interview (two responses per facilitator; one for each learning cycle)
Youth focus groups	Instrument 4. Youth focus group protocol	Respondents: Youth program participants Content: Experiences with program, understanding of mediators associated with the selected strategy, and overall satisfaction Purpose: To gauge participants’ perceptions and determine whether further revisions could improve the strategy	Mode: In person or videoconference Duration: Up to 45 minutes

Other Data Sources and Uses of Information

Site selection will be informed by grantee program plan data collected through the SRAE National Evaluation (OMB Control Number 0970-0530, approved July 2019) as well as the applications of grantees funded after that data collection effort.

To evaluate training and implementation of these new strategies, the study team will also review the responses to youth program entry and exit surveys that the sites are required to administer and report to ACF (OMB Control Number 0970-0536, approved August 2020).

A3. Use of Information Technology to Reduce Burden

The study team plans to use information technology wherever possible. The study team plans to conduct youth focus groups and facilitator interviews in person but is prepared to conduct both of these data collection activities remotely—via videoconference if necessary—because of the COVID-19 pandemic. With participant permission, the study team also plans to record focus groups and interviews to help reduce respondent burden by reducing the time needed for the data collection by eliminating note taking during the data collection. Regardless of circumstances, the facilitator survey will be available as an online survey. We will provide a link via email that facilitators can use to access and complete the survey using a tablet, smartphone, or laptop.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The data necessary to conduct the proof-of-concept studies and assess the implementation of the two facilitation strategies are not collected or available elsewhere. As discussed in A2, training methods will be newly developed or applied for this proof-of-concept. None of the instruments ask for information that can be reliably obtained through other sources.

A5. Impact on Small Businesses

The programs participating in the study will be small, non-profit organizations. The SRAENE team will request information required only for the intended use. The burden for respondents will be minimized by restricting the interview and survey length to the required minimum, conducting interviews at times convenient for the respondents, and not requiring additional record-keeping on the part of the programs.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 13, 2020, Volume 85, Number 198, page 64480, and provided a 60-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

Several experts in SRAE programming and research provided consultation to the study team and ACF on many occasions between 2018 – 2020. Feedback received from experts identified the need for this formative study. Staff from HCS will develop training content related to the Heritage Method and members of the SARHM team will assist with training materials for the co-regulation strategy.

A9. Tokens of Appreciation

No tokens of appreciation are planned for this data collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

This data collection effort will collect personally identifiable information (PII) from program facilitators (names, work email addresses, and telephone numbers) and program participants (names, email addresses, and telephone numbers) to obtain consent to participate in data collection activities and arrange data collection (including scheduling and sending invitations/reminders to data collection activities).

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individual’s personal identifier.

Assurances of Privacy

All study participants will be informed of the planned uses of data, that their participation is voluntary, and that the study team will keep their information private to the extent permitted by law. The study team will discuss issues of privacy during training sessions with staff working on the project. The contractor, Mathematica, requires that staff complete online security awareness training when they are hired and then participate in annual refresher training thereafter. Training topics include the security policies and procedures outlined in the Mathematica Corporate Security Manual. Program staff will transfer records to Mathematica by using a secure file transfer protocol site in case the files contain personally identifiable information. As specified in the contract, Mathematica will protect respondent’s privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information.

Parent consent and youth assent forms (Appendix B. Parent Consent and Youth Assent Forms) inform parents and youth that they are invited to participate in focus groups, that the information requested from them is for program improvement purposes only, and that their identities will not be disclosed to anyone outside the study team. With participants’ permission, focus groups will be recorded, and participants will be assured that their recorded comments will be saved only until transcribed and that the transcription summaries will not reveal their identities. All participants (and their parent or legal guardian) must read and acknowledge the form before participating in the data collection. The study will be reviewed by Mathematica’s Institutional Review Board, Health Media Lab. Outreach and data collection will not begin until IRB approval has been received.

Data Security and Monitoring

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security Plan that assesses all protections of respondents’ PII. The contractor will ensure all of its employees, subcontractors (at all tiers), and employees of each subcontractor who perform work under this contract and subcontract receive training on data privacy issues and comply with the above requirements. All Mathematica staff must sign an agreement to maintain the privacy of any information from individuals, businesses, organizations, or families participating in any projects conducted by Mathematica, complete online security awareness training when they are hired, and participate in a refresher training annually.

As specified in the evaluator’s contract, the contractor will use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Information Processing Standard. The contractor will incorporate this standard into its property management and control system and establish a procedure to account for all laptop and desktop computers and other mobile devices and portable media that store or process sensitive information. The contractor will secure any data stored electronically in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing to the extent possible including sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive information that ensures secure storage and limits on access.

No information will be given to anyone outside the SREANE study team and ACF. All PII, typed notes, and audio recordings will be stored on restricted, encrypted folders on Mathematica’s network, which is accessible only to the study team.

A11. Sensitive Information

There are no sensitive questions as part of the facilitator surveys or interviews (Instruments 1, 2, and 3).

The youth focus group protocol (Instrument 4) will include some questions about youth impressions of the SRAE programs, their overall reaction to the facilitation strategies used, and their relationships with staff and peers which some program participants might consider sensitive. However, these questions are essential for understanding how facilitators are using the strategies. The study team will obtain active consent in all sites and will inform potential study participants of the purpose of the data collection, stressing that participants may refuse to answer any question. Additionally, the protocol and all related materials, such as the consent form, will have been approved by an institutional review board.

A12. Burden

Explanation of Burden Estimates

1. Facilitator Pre-Training Survey (Instrument 1): Facilitators participating in each study will participate in one training session on the facilitation strategy before the start of the first learning cycle. Across the two studies, the study team will administer a 10-minute web-based survey to up to 36 facilitators participating in training on one of the strategies across a total of nine sites. The team will administer the pre-training survey before training for a total of 36 surveys (36 facilitators * 1 survey).

2. Facilitator Post-Training Survey (Instrument 2): The post-training survey will be administered after the trainings among the same population as the pre-training survey, for a total of 36 surveys (36 facilitators * 1 survey). This web-based survey is also expected to take 10 minutes.

3. Facilitator interviews (Instrument 3): The study team will administer a 45-minute semi-structured interview protocol with up to 36 facilitators who are using one of the strategies across a total of nine sites. The team will administer the interview with staff twice, once during each learning cycle, for a total of 72 interviews (36 facilitators * 2 interviews).

4. Youth focus groups (Instrument 4): Assenting program participants with parent consent will participate in focus groups, with one focus group per school. Among up to nine sites across both studies, we expect that each site will operate in four schools each with two classrooms of 25 students, for a total of 1,800 students. Of the 1,800 students, we estimate up to 20%, or 360 students, will return signed consent forms and will show up to participate in 45-minute focus groups (9 sites * 4 schools * 2 classrooms * 25 students * 1 learning cycle * 20% participation). Focus groups will be restricted to one per school (36 schools), each with a total of up to 10 participants. If more than 360 students agree to participate in focus groups, we will purposefully sample among the classrooms so as not to exceed the 360 total participants (see Supporting Statement Part B, Section B2).

Estimated Annualized Cost to Respondents

To compute the total estimated annual cost, we multiplied the total annual burden hours by the average hourly wage for each adult participant, based on median weekly wages from the fourth-quarter estimates of the Current Population Survey, Bureau of Labor Statistics (2021). We present the results in Table A.12. According to the Bureau of Labor Statistics’ Current Population Survey 2020², the average hourly wage for full-time community and social service specialists (facilitators) is $25.09.

Table A.12. Estimated annualized cost to respondents

Instrument	Number of respondents (total over request period)	Number of responses per respondent (total over request period)	Average burden per response (in hours)	Total /annual burden (in hours)	Average hourly wage rate	Total annual respondent cost
Instrument 1. Facilitator Pre-Training Survey	36	1	0.167	6	$25.09	$150.84
Instrument 2. Facilitator Post-Training Survey	36	1	0.167	6	$25.09	$150.84
Instrument 3. Facilitator Interview Guide	36	2	0.75	54	$25.09	$1,354.86
Instrument 4. Youth Focus Group Protocol	360	1	0.75	270	$0	$0
Estimated annual burden total				336		$1,656.54

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total estimated cost for the federal government for the data collection activities under this current request is $720,186, including personnel effort plus other direct and indirect costs.

Cost category	Estimated costs
Field work	$680,140
Publications/dissemination	$40,046
Total costs over the request period	$720,186

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

A16. Timeline

The information collected under this request will be used to learn about the implementation of two facilitation strategies at selected programs for the SRAENE Program Components Impact study. Over a three month period, the study team will identify implementation challenges and strategies and prepare for the future possible summative evaluations (that will be covered under a forthcoming full request). Beginning in January 2022, pending OMB approval of the Generic request, the study team will work with selected sites to implement one of two facilitation strategies and initiate data collection activities. The study team will develop an internal memo for ACF summarizing learning from the proof-of-concept study, describing how sites implemented the strategies and what lessons were learned to inform a potential future summative evaluation. ACF may also consider producing a brief summarizing the strategies and lessons learned from the pilot.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendix A. Facilitator Survey Invitation Email

Appendix B. Parent Consent and Youth Assent Forms

Instrument 1. Facilitator Pre-Training Survey

Instrument 2. Facilitator Post-Training Survey

Instrument 3. Facilitator Interview GuideInstrument 4. Youth Focus Group Protocol

1 Baumgartner, S., Frei, A., Paulsell, D., Herman-Stahl, M., Dunn, R., and Yamamoto, C. (2019). SARHM: Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Programs for Youth. OPRE Report #2020-122. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

2 https://www.bls.gov/oes/current/oes_nat.htm

9