SSB Res Eval GenIC - Sexual Risk Avoidance Education National Evaluation: Co-regulation Guidance Pilot Study

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Sexual Risk Avoidance Education National Evaluation: Co-regulation Guidance Pilot Study

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

January 2025

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Tia Brown

Kathleen McCoy

MeGan Hill

Nakia Martin-Wright

Part B

B1. Objectives

Study Objectives

The Sexual Risk Avoidance Education National Evaluation (SRAENE) core components impact study (CIS) seeks to refine and test improvements to one or more program components Sexual Risk Avoidance Education (SRAE) programs to ultimately improve youth outcomes related to sexual risk avoidance, healthy relationships, and well-being. The SRAENE team is currently focused on improving the component of facilitation by training facilitators to use a set of six co-regulation strategies designed to support youth self-regulation. This request is specific to a co-regulation guidance pilot study that aims to: (1) gather evidence to inform the direction of a future large-scale evaluation of the effectiveness of using co-regulation strategies, (2) assess the feasibility of facilitators consistently implementing the co-regulation strategies, and (3) help to further ACF’s co-regulation learning agenda.

Specifically, the co-regulation guidance pilot study seeks to document the feasibility of using guidance that describes where facilitators should integrate the co-regulation strategies into a SRAE curriculum. This builds on prior work in using co-regulation facilitation strategies (see proof of concept pilot study, OMB#: 0970-0531; formative evaluation for the program components impact study, OMB#: 0970-0356). This prior work suggested promise, but required further information about how to support facilitators with implementing the strategies consistently before conducting a larger scale replication and evaluation. Supporting Statement Part A, Section A1 provides the background for this phase of the study, and Section A2 discusses the purpose and use of the data.

Generalizability of Results

This study is not meant to promote statistical generalization to other program, sites, or service populations. Data collected under this generic information collection request will be used to refine SRAE program delivery strategies, furthering ACF research focused on this area.

Appropriateness of Study Design and Methods for Planned Uses

As discussed in Supporting Statement Part A, Section A2, this pilot study is designed to further learn about the feasibility of using co-regulation integration guidance to promote consistent use of the co-regulation strategies in SRAE classes. We will assess the implementation of the co-regulation strategies and use of the guidance to understand the successes and challenges associated with broader implementation of the strategies to build youth’s self-regulation skills. These co-regulation strategies have been used in nine SRAE programs in the formative evaluation and showed promise. However, further information about consistent implementation is needed before larger-scale replication through an impact study. Thus, this pilot study is designed to further research conducted under the formative evaluation by assessing the use of an additional training and support tool (the integration guidance). We will use a daily facilitator log (Instrument 1) to track how often facilitators are using the strategies during a lesson, which will help demonstrate whether facilitators are using the strategies consistently. We will use brief, pulse check surveys (Instruments 2 and 3) after training and coaching to determine how satisfied facilitators are with the training and support they receive and determine if there are outstanding gaps that need to be addressed. Finally, we will use the brief mini-interviews (Instrument 4) to capture facilitators feedback more broadly and their perceptions of how feasible it is to use the strategies and the associated guidance while facilitating. These quick, consistent check-ins will allow us to observe how facilitators’ experiences may change over time as they get more comfortable with the strategies. Data collected in this study are not intended to be representative.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The target populations for this generic information collection request are facilitators of SRAE programs in sites that deliver SRAE programs in-school, during the school day. To identify the target populations, we will begin with the SRAE grant recipients and their sub-recipient providers. Eligible sites are those that (1) deliver a selected SRAE curriculum (Love Notes, which is the most prevalent curriculum across SRAE grant recipients¹), (2) implement programming with high school aged students in schools, and (3) implement programming during the school day. These sites will be identified through a process that includes a review of their program plans and discussions between ACF and the contractor conducting this work, Mathematica.

Sampling and Site Selection

ACF will invite grant recipients meeting the eligibility criteria described above to volunteer to be a part of this study. ACF will prioritize selecting grant recipients to be one of the two sites where facilitators have experience delivering the selected SRAE curriculum.

The study team anticipates that each of the two sites will employ approximately 5 facilitators, for a total of 10 facilitators in the study. These assumptions are based on a review of grant recipient applications. We expect all facilitators within a site to participate in the study.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The instruments for this data collection request are informed by both the proof of concept pilot study and the formative evaluation for the program components impact study. The facilitator log (Instrument 1) is adapted from a similar instrument used in the formative evaluation for the program components impact study to learn about facilitators’ adherence to the co-regulation strategies and their perceptions on how using the strategies went. The instrument used in the formative evaluation had very low response rates and high amounts of missing data. Based on this, we revised the instrument to be shorter and easier to answer quickly to help promote high response rates. The post-training and post-coaching pulse check surveys (Instruments 2 and 3) and the facilitator mini-interview (instrument 4) are new instruments, not tested during the prior formative evaluation. However, items in each instrument are informed by the prior formative evaluation. For example, when developing the facilitator mini-interview guide, we created a list of common reasons why strategies had or had not gone well for facilitators based on findings from the previous studies where during longer interviews facilitators described what supported and hindered their use of the strategies.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection. Two Mathematica study team members (one to lead coaching; one to lead data collection) will be assigned to each site to support activities. Each team will conduct or oversee survey administration and qualitative data collection with the SRAE program facilitators. To ensure an efficient and standardized data collection process, Mathematica study staff will participate in a project-specific data collection training. The training will cover the expectations and process for collecting information from the facilitators and cover all aspects of the data collection. All staff will be trained on best practices for collecting high-quality data and procedures around data privacy and security.

The program facilitators will participate in training on the co-regulation strategies (described in Supporting Statement A, Section A2). Immediately following the last training session, facilitators will complete the post-training pulse check survey (Instrument 2). Facilitators will participate in bi-weekly coaching sessions with a Mathematica study team member to support their use of the strategies. Immediately following the last coaching session, facilitators will complete the post-coaching pulse check survey (Instrument 3). During a 6-week period, facilitators will complete a log (Instrument 1. Facilitator log) that asks which strategies they used and how they thought using each strategy went at the end of each lesson they teach. At the end of each week (for the same 6-week span), facilitators will be asked to participate in a brief, one-on-one, semi-structured interview (Instrument 4. Facilitator mini-interview guide).

Table B.1. lists all data collection activities proposed for the co-regulation guidance pilot study.

Table B.1. SRAENE Co-regulation guidance pilot study data collection activities

Data Collection	Administration plans
Facilitator log	Total participants	10
	Mode	Web
	Time	2 minutes
	Frequency	Daily, during selected weeks
Facilitator Post-Training Pulse Check	Total participants	10
	Mode	Web
	Time	5 minutes
	Frequency	1
Facilitator Post-Coaching Pulse Check	Total participants	10
	Mode	Web
	Time	5 minutes
	Frequency	1
Facilitator Mini-Interview	Total participants	10
	Mode	Conference call
	Time	15 minutes
	Frequency	Weekly, during selected weeks

Data collection activities. The study team will use a web-based survey platform to collect data for the facilitator log and pulse check surveys from all program facilitators. The facilitators will receive email invitations containing a link to their secured surveys (Appendix A. Facilitator Survey Invitation Email). For the facilitator log, facilitators can elect to complete it on paper and then complete an electronic form of the log to submit it at the end of each day, or complete the log directly into the electronic form.

The facilitator mini-interviews will take place weekly over 6 weeks in the spring 2025 semester. The interview data will be collected via conference call on a virtual platform such as Webex or Zoom. The study team will work with the facilitators to schedule the interview and provide a link to the conference call. The interviewer will begin the interview by reminding the participant of the expectations and purpose of the interview and requesting permission to record the interview. The participants will then engage in a guided discussion about what strategy went well that week and what strategy was challenging. The interviewer will take notes during the interview and interviews will be audio recorded to ensure accurate collection of data. The facilitator will cross-check their notes against the recorded interview to ensure accuracy and completeness.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The surveys and qualitative data collection activities are not designed to produce statistically generalizable findings, and participation is at the respondent’s discretion.

NonResponse

Participants will not be randomly sampled, and findings are not intended to be representative. Consequently, we will not calculate nonresponse bias. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Data collected for this pilot study will document the implementation of the co-regulation strategies and use of newly developed integration guidance to support facilitators’ use of the strategies in the classroom. These efforts will work to build evidence for promising practices related to the feasibility of training and implementing this facilitation strategy, critical for furthering ACF’s research agenda on co-regulation, assessing the need for a future rigorous study of the strategies, and supporting technical assistance to the SRAE grant recipients. The data will not be used to generate population estimates, either for internal use or for dissemination.

B7. Data Handling and Analysis

Data Handling

No personally identifiable information (PII) will be shared outside of the study team. Survey data and qualitative data, including typed notes and audio recordings, will be stored on Mathematica’s secure network, which is accessible only to the study team, and destroyed at the end of the study. Each facilitator will receive an ID code and all data will be saved under the ID code, rather than using their names. Transcripts from facilitator mini-interviews will be de-identified to remove PII. Any information linking the ID code and facilitators’ PII will be saved on Mathematica’s secure restricted drive and be password protected.

Data Analysis

This project will not employ complex data analytic techniques. To analyze qualitative data—such as notes from the facilitator interviews—we will use standard qualitative analysis techniques such as thematic identification. For the facilitator surveys, we will conduct standard qualitative analysis of responses to open-ended items; and calculate ranges, averages, and simple descriptive statistics for the quantitative questions.

Data Use

Data collected will be used to assess the implementation of the co-regulation strategies and the feasibility of using the integration guidance, including describing how the facilitators use the strategies in their instruction, identifying whether the guidance was helpful and allowed them to implement the strategies with fidelity, and determining whether facilitators need any additional supports.

The findings from this pilot study will inform the planning of a possible future effectiveness evaluation of the co-regulation strategies². The primary purpose of the information collected is not publication, but findings could be incorporated into materials that are made publicly available. For example, to contextualize the plans for successful replication for a future study, ACF may reference the findings from this pilot study phase.

B8. Contact Person(s)

In Table B.2, we list the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Individuals Responsible for Study

Name	Affiliation	Email address
Calonie Gray	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Kathleen McCoy	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
MeGan Hill	Family and Youth Services Bureau Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Susan Zief	Mathematica	[email protected]
Heather Zaveri	Mathematica	[email protected]

Attachments

Appendix A. Study Notification and Reminder Materials

Instrument 1. Facilitator log

Instrument 2. Facilitator post-training pulse check

Instrument 3. Facilitator post-coaching pulse check

Instrument 4. Facilitator mini-interview guide

1 Neelan, T., DeLisle D., & Zief, S. (2022). The Title V Competitive and General Departmental Grantees’ Sexual Risk Avoidance Education Program Plans (OPRE Report No. #2022-91). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

2 Related information collections that are subject to the Paperwork Reduction Act will be submitted for review and approval under an appropriate mechanism (generic or full approval) as needed.

10