Generics SSB - CIS Proof of Concept

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Sexual Risk Avoidance Education National Evaluation

Program Components Impact Study:

Proof of Concept Pilot Phase

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

November 2021

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Jessica Johnson

Part B

B1. Objectives

Study Objectives

The objectives of this proposed generic information collection (GenIC) are to test and refine training on and implementation of two facilitation strategies that aim to improve the delivery of SRAE programs to youth and determine the feasibility of a future summative evaluation. To meet these objectives, the study will use rapid-cycle learning (RCL) and collect data from selected Sexual Risk Avoidance Education (SRAE) programs through facilitator surveys, facilitator interviews, and youth participant focus groups. The data collected from these proof-of-concept studies will be used to to develop and refine training on and implementation of the two program delivery strategies and to determine if a summative evaluation is feasible. If a summative evaluation is determined to be feasible, we will submit a full information clearance request to the Office of Management and Budget (OMB).

Generalizability of Results

This study is intended to present an internally valid description of the feasibility of implementing a strategy in the chosen grantee programs and is not meant to promote statistical generalization to other programs or service populations. Data collected under this GenIC will be used to refine the program delivery strategies and assess the feasibility of conducting summative evaluations of the strategies in fall 2022.

Appropriateness of Study Design and Methods for Planned Uses

As discussed in Supporting Statement Part A, Section A2, the proof-of-concept studies are designed to strengthen and test two promising SRAE facilitation strategies: (1) a set of six co-regulation strategies to support facilitators with building youths’ self-regulation skills, and (2) the Heritage Method, which teaches facilitators to assess youths’ attitudes and beliefs on constructs associated with the delay of sexual initiation. Both strategies have been used in prior SRAE programs, but the implementation has been limited to the programs that supported their development.

As discussed in Supporting Statement Part A, Section A2, the proposed proof-of-concept studies and associated data collection use RCL study designs to quickly learn about the viability of training SRAE program facilitators on the two selected strategies and assess whether either or both strategies could be expanded for a future summative evaluation of each strategy. The primary purpose of the data collection under this GenIC is to collect feedback on the two SRAE facilitation strategies being tested across up to nine sites. The RCL data collection method is ideal for this proposed study because it allows for iterative data collection to inform immediate technical assistance and implementation needs associated with employing each strategy. Facilitators working in nine sites will be trained to implement one of two strategies in their SRAE programs. During the implementation period, the study team will conduct two RCLs at each site. This iterative data collection strategy is designed to collect initial feedback on the use of the strategies (Learning Cycle 1), use that feedback to identify opportunities to strengthen and refine the strategy, and then implement and collect feedback on the refined strategy (Learning Cycle 2). Details about the specific data collection activities are described in Section B2.

As noted in Supporting Statement Part A, this information is not intended to be used as the principal basis for public policy decisions, and it is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target populations for this generic information collection request are facilitators of SRAE programs and youth receiving SRAE programs in-school, during school. To identify the target populations, we will begin with the universe of direct service organizations receiving SRAE grants from ACF’s Family and Youth Services Bureau, such as those funded through the General Departmental and Competitive grant programs. Eligible sites are those that deliver SRAE programming in schools with professional facilitators and that implement curricula aligned with the objectives of each facilitation strategy to be tested. These programs will be identified through a process that includes a review of their program plans, including data previously collected by the study team (OMB Control Number 0970-0530, approved July 2019). An early review of these data suggest that there will be approximately 50 eligible sites.

Sampling and Site Selection

We will recruit up to nine SRAE program sites to participate in these proof-of-concept studies. During the recruitment process, we will reach out to sites meeting the criteria described above to determine interest and capacity to participate in the proof-of-concept studies. For example, sites must have SRAE program facilitators available to participate in the trainings and have plans to implement programming during the two RCL periods scheduled for early 2022. We will select a small number of sites to implement (four or five for each strategy, with no more than nine total), which is consistent with a RCL design and to efficiently and quickly gather the data to inform possible future summative evaluations.

The study team anticipates that each of the nine possible sites across the two studies will employ approximately 36 facilitators and will operate in four schools with two classrooms of 25 students in each school (for a total of 1,800 students). These assumptions are based on a review of grantee applications and previous data collected for the SRAE National Evaluation (OMB Control Number 0970-0530, approved July 2019). These 36 facilitators and their students comprise the eligible sample for the studies.

All 1,800 youth program participants in the facilitators’ classrooms will be eligible to participate in the studies, provided they assent and have parental consent to participate in the focus groups. We assume that 20 percent of youth will have parental consent and will show up to participate in the focus groups. We anticipate that youth program participants will be in middle and high school and between the ages of 12 to 18. Program participants involved in data collection will be from a convenience sample; they may not be representative of the population all SRAE programs serve.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The study team adapted the instruments from similar RCL protocols and surveys used for other ACF-funded studies, including projects to examine educator training for youth-serving Healthy Marriage and Relationship Education grantees and to strengthen the implementation of two-generation programs serving parents and children in the same family.¹

The instruments are purposefully broad to allow for flexibility across programs and respondents. The facilitator surveys (Instruments 1 and 2) collect knowledge-based information from the facilitator training through a pre- and post-training survey, and the facilitator’s interview topic guide (Instrument 3) covers topics related to use of the strategy, perceived effectiveness of the strategy, and suggestions for improvement. The youth focus group protocol (Instrument 4) covers a range of topics related to youth feedback on the course and the facilitation strategies, and their own and other students’ engagement and interest in the classes and topics covered.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection. Three Mathematica study team members (one lead and two supports) will be assigned to each of the nine sites to support the RCL data collection activities. Each team of three will conduct or oversee surveys and qualitative data collection with the SRAE program facilitators and the youth program participants.

The frontline program facilitator staff and their supervisors will participate in training on one of the new strategies (the Heritage Method or co-regulation, described in Supporting Statement A, Section A2). At the start of training, facilitators and their supervisors will complete a pre-training web-based survey and following training they will complete a post-training web-based survey (Instrument 1. Facilitator Pre-Training Survey and Instrument 2. Facilitator Post-Training Survey) and semi-structured in-person or virtual interviews (Instrument 3. Facilitator Interview Guide). Based on the feedback received, the study team will partner with the sites to identify and implement solutions tailored to the context and specific challenges of each site and conduct a series of learning cycle assessments of the solutions. As described in Section B1, data collection activities will occur over two RCLs. During each learning cycle a variety of data will be collected, including SRAE program facilitator interviews and youth program participant focus groups (Instrument 4. Youth Focus Group Protocol).

Table B.1 lists all data collection activities proposed for this proof-of-concept study.

Table B.1. SRAENE Proof-of-Concept study: Data collection activities

Data Collection	Administration plans
Facilitator Pre-Training Survey	Facilitation Strategy	Heritage Method and Co-Regulation
	Total participants	36
	Mode	Web
	Time	10 minutes
Facilitator Post-Training Survey	Facilitation Strategy	Heritage Method and Co-Regulation
	Total participants	36
	Mode	Web
	Time	10 minutes
Facilitator Interview	Facilitation Strategy	Heritage Method and Co-Regulation
	Total participants	36
	Mode	In-person, semi-structured interview protocol
	Time	60 minutes
Student Focus Groups	Facilitation Strategy	Heritage Method and Co-Regulation
	Total participants	360
	Mode	In-person or virtual, focus group protocol
	Time	45 minutes

Facilitator data collection activities. The study team will use a web-based survey platform to collect the pre- and post- training survey data from all program facilitators. The facilitators will receive email invitations containing a link to their secured surveys (Appendix A. Facilitator Survey Invitation Email). The pre-training survey will take place just before the training begins, and the post-training survey will take place immediately at the end of training. The training staff will help the facilitators access and provide support while they complete their online surveys prior to leaving the training, ensuring a 100 percent completion rate.

The facilitator interviews will take place once within each of the two learning cycles, with the first interview taking place at the end of learning cycle 1 and the second interview occurring at the end of learning cycle 2. The interview data will be collected in person during a site visit by two study team members. The interview can also be conducted virtually if necessary. The study team will work with the facilitators ahead of the site visit to schedule the interview time and location.

Data collection from youth program participants. The youth program participants will have the opportunity to participate in a focus group to discuss both their impressions of the program and its facilitation. Program facilitators will send paper versions of consent forms for parents and guardians home with youth during the second learning cycle (Appendix B. Parent Consent and Youth Assent Forms). While the rapid nature of the evaluation activities prevents the study team from conducting parent reminders or other consent form nonresponse follow-up activities, we expect that the proposed approach of sending consent forms to all students within each selected classroom will yield sufficient numbers of returned and consented forms to allow for a focus group of 10 diverse youth participants per school.

The youth focus groups will be conducted in all study sites, and youth will be selected into the focus groups based on both their parent’s consent to participate and their own assent, and other factors as discussed in Section B2. The focus groups will be conducted in person on the school grounds, during the school day (e.g., during students’ lunch period), but can be conducted virtually if necessary. Two study team members will conduct each focus group, with a senior member of the team moderating and a junior member taking notes.

All study team members will receive training to ensure that any data collected from the facilitators and the youth are collected in a consistent and high-quality manner. With participant permission, virtual interviews and focus groups will be recorded, with a junior staff member taking notes during all of them. Lastly, to ensure quality and consistency, the team will meet often to discuss program activities and troubleshoot issues as they arise.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The surveys and qualitative data collection activities are not designed to produce statistically generalizable findings, and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

Nonresponse

Participants will not be randomly sampled, and findings are not intended to be representative. Consequently, we will not calculate nonresponse bias. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Data collected for the proof-of-concept studies will support decision-making on the feasibility of pursuing a summative evaluation of either or both facilitation strategies, guide decisions regarding improving one or both strategies, and build evidence for promising practices related to the feasibility of training facilitators on two facilitation strategies. The data will not be used to generate population estimates, either for internal use or for dissemination.

B7. Data Handling and Analysis

Data Handling

No personally identifiable information will be given to anyone outside of the study team. Survey data and qualitative data, including typed notes and audio recordings, will be stored on Mathematica’s network, which is accessible only to the study team, and destroyed at the end of the study.

Data Analysis

This project will not employ complex data analytic techniques. To analyze qualitative data—such as notes from the facilitator interviews and youth participant focus groups—we will use standard qualitative analysis techniques such as thematic identification. For the facilitator surveys, we will conduct standard qualitative analysis of responses to open-ended items; and calculate ranges, averages, and simple descriptive statistics for the quantitative questions.

Data Use

The study team, working with the sites selected to participate in the pilot, will use the data collected during the RCL process to inform improvements on how the sites can refine and use the strategies in their programs. In addition, the study team will release an internal memorandum to FYSB describing knowledge gained about common implementation challenges and solutions. The memorandum will provide details about the design and general insights, lessons, and themes from conducting RCLs of the SRAE facilitator strategies across the multiple program sites. ACF may also consider producing a brief summarizing the strategies and lessons learned from the pilot for SRAE grant recipients.

The products will not share quantitative findings about the programs. In sharing findings, we will describe the study methods and limitations to generalizability and as a basis for policy.

B8. Contact Person(s)

In Table B.2, we list the federal and contract staff responsible for the study, including each individual’s affiliation and email address.

Table B.2. Individuals responsible for study

Name	Affiliation	Email address
Calonie Gray	Office of Planning, Research, and Evaluation, Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Jessica Johnson	Family and Youth Services Bureau, Administration for Children and Families U.S. Department of Health and Human Services	[email protected]
Susan Zief	Mathematica	[email protected]
Heather Zaveri	Mathematica	[email protected]
Tiffany Waits	Mathematica	[email protected]
Kim McDonald	Mathematica	[email protected]

Attachments

Appendix A. Facilitator Survey Invitation Email

Appendix B. Parent Consent and Youth Assent Forms

Instrument 1. Facilitator Pre-Training Survey

Instrument 2. Facilitator Post-Training Survey

Instrument 3. Facilitator Interview Guide

Instrument 4. Youth Focus Group Protocol

1 Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM, OMB Clearance #0970-0355, approved June 2018); Next Steps for Rigorous Research on Two-Generation Approaches (OMB Clearance #0970-0356, approved July 2020); and Strengthening the Implementation of Marriage and Relationship Programs (SIMR, OMB Clearance #0970-0531, approved October 2021).

7