Supporting Statement B - GenIC Formative for Program Support - SIMR

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Marriage and Relationship Programs (SIMR)

Formative Data Collections for Program Support

0970-0531

Supporting Statement

Part B

March 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Shirley Adelstein

Part B

B1. Objectives

Study Objectives

For many years, the Office of Planning, Research, and Evaluation (OPRE) in the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (DHHS) has led a sustained effort within the federal government to develop, document, and evaluate HMRE programs, particularly those serving populations with low incomes. SIMR seeks to generate evidence about how to improve the implementation of HMRE programs in preparation for summative evaluation by testing and refining strategies to overcome common implementation challenges related to recruitment, retention, and engagement. The study team will partner with HMRE programs to implement solutions tailored to the context and specific challenges of each program and conduct multiple learning cycle assessments of the solutions. Early cycles will seek to understand the feasibility of implementing a strategy, while later cycles will examine the success of the solutions through more rigorous methods.

The purpose of this proposed generic information collection GenIC is to complete rapid cycle learning (RCL) activities in selected programs to strengthen programs in preparation for potential summative evaluation. This request includes a staff survey (Attachment A), a participant focus group protocol (Attachment B), and a staff interview topic guide (Attachment C).

Generalizability of Results

This study is intended to present an internally-valid description of the feasibility of implementing a strategy in the chosen programs, not to promote statistical generalization to other programs or service populations.

Appropriateness of Study Design and Methods for Planned Uses

These data collection efforts are intended to support RCL by informing decision-making and technical assistance for individual programs. In RCL, programs iteratively implement a strategy designed to address an implementation challenge, collect feedback on the use of the strategy, use the feedback to identify opportunities to strengthen and refine the strategy, and then implement and collect feedback on the refined strategy. The primary purpose of the data collection efforts are to collect feedback on a given strategy that a given program is testing.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

Participation in this study is restricted to HMRE programs funded by ACF’s Office of Family Assistance in September 2020. The study team will select 10 SIMR programs and from these programs, we will use the data collection instruments to collect information from program staff and program participants.

• Program staff. Program directors, managers, supervisors, and frontline staff will participate in semi-structured interviews (Attachment C). Frontline staff will also respond to web surveys (Attachment A). The program staff asked to respond will include those directly involved in implementing and testing a strategy using RCL.

• Program participants. Program participants will be asked to participate in virtual focus groups (or interviews, if focus groups are infeasible; see A2) (Attachment B). Participants in HMRE programs include youth (as young as 14) and adults. They may be married, in a committed romantic relationship, or not currently partnered. Many are from racial and ethnic minority backgrounds, are pregnant or parenting, or experiencing economic disadvantage, such as low incomes and limited education. Programs being considered for SIMR include those serving rural and urban geographies, participants in corrections centers, Spanish-speakers, pregnant or parenting youth, and predominantly African American communities (see below). Program participants involved in data collection will be from a convenience sample; they may not be representative of the population that the programs serve. We will aim to capture feedback from clients with a range of characteristics and experiences to understand how a strategy works, under what circumstances, and for whom.

Sampling and Site Selection

Sites selected to participate in SIMR include 10 programs that offer programming aligning with ACF learning priorities (for example, serving at-risk, marginalized, and/or understudied populations), that could identify a challenge relevant to SIMR’s priority approaches, and that have the capacity to engage in RCL. These programs were identified through a process that included a review of grantee applications and findings from the previously approved SIMR information collection (OMB Control Number 0970-0356, approved October 2020).

The 10 programs selected for SIMR include five programs serving adult populations and five serving youth.

Anthem Strong Families: Through the Family Champions Project, the grantee will offer TYRO Leadership and CORE Communications to individual adults – with a focus on Hispanic, single parents, and those with low incomes. Anthem Strong Families is planning to offer virtual services in April and to offer both in-person and virtual service options when safe to do so.

Family Service Agency of Santa Barbara County (Santa Barbara, California). Through the Healthy Marriage and Relationship Stability Program (HMRESP), the grantee plans to offer Within My Reach and PREP supplemented with additional material on parenting. The grantee will run separate groups for individual adults and couples and will serve a primarily Hispanic population. Family Service Agency of Santa Barbara County expects to be providing virtual services in April and will continue offering them as an option throughout their grant. They will start to provide in-person services when it is safe to do so.

Gateway Community Services (West Liberty, Kentucky). Through their FRAMEWorks program, the grantee will offer Within My Reach supplemented with job and career advancement content to individual adults across nine rural counties in eastern Kentucky. The grantee is an Early Head Start/Head Start provider and plans to recruit participants primarily from these programs. Gateway Community Services will be offering virtual services in April and will return to in-person services when it is safe to do so.

Montefiore Medical Center (Bronx, NY). Through the Supporting Healthy Relationships Program, the grantee will offer the PREP and Bringing Baby Home curricula integrated with job and career advancement services. Montefiore Medical Center is planning to offer two versions of its program, one in person and one virtual, throughout the grant period.

The Ridge Project (McClure, Ohio). Through their TYRO Couples program, the grantee plans to offer the TYRO Couples curriculum supplemented with job and career advancement services to incarcerated men and women who are married or in a romantic relationship and about to be released from prison. Services will be offered in nine Ohio prisons. Partners are invited to attend. The Ridge Project will start by providing virtual services in April and will return to in-person services when it is safe to do so.

Auburn University (11 counties, Alabama). Through the Alabama Youth Relationship Education (AYRE) program, Auburn University plans to provide HMRE services to youth in grades 9-12 in high school and community settings in 11 counties across the state. AYRE includes Relationship Smarts Plus, Mind Matters, and Money Habitudes. Facilitators will be either trained adult HMRE educators employed by community organizations or “near peers,” who are trained undergraduate students. Auburn expects to be providing virtual services in April. They will shift to in-person services beginning in the fall depending on COVID-19.

More than Conquerors, Inc. (MTCI, suburban Atlanta, Georgia). Through MATURE Plus III, MTCI plans to serve youth in grades 9-12 in three suburban Atlanta high schools with REAL Essentials Advance. The curriculum will be supplemented with case management. The program will be provided in required health classes. MTCI expects to be providing virtual services in April. They will shift to in-person services beginning in the fall depending on COVID-19.

Youth and Family Services (Rapid City, South Dakota). Youth and Family Services plans to offer Relationship Smarts Plus to youth in a number of traditional and alternate high schools in Rapid City, South Dakota, and the surrounding area, which includes American Indian reservations and rural areas. The schools have a high proportion of American Indian students. Workshops will be provided weekly in high schools and three times per week in alternate educational settings. Youth and Family Services expects to be providing virtual services in April. They will shift to in-person services beginning in the fall depending on COVID-19.

University of Denver (Denver, Colorado). The grantee plans to adapt its MotherWise program for pregnant and parenting young women in the Denver area. Most of these young mothers will be identified through a partnership with Denver Health, the largest hospital system in the Denver area. The program plans to use Love Notes as the foundation of its curriculum, offered in English and Spanish in six weekly workshops. University of Denver will be providing virtual services in April and will shift to in-person services when it is safe to do so, depending on the availability of COVID-19 vaccines for pregnant women and new mothers.

Texas A&M Agrilife Extension (Central Texas). Success…Powered By You plans to serve 18-to-24-year olds in five rural counties in the Brazos Valley in Texas. They plan to offer Relationship Smarts Plus over 4-to-5 weeks. The programs will also offer two optional additional curricula: (1) a six-hour, weekly parenting curriculum, and (2) a four-hour, weekly financial education curriculum. Participants will also receive case management. Texas A&M expects to be providing virtual services in April and will continue offering them as an option throughout their grant. They will start to provide in-person services when it is safe to do so.

To recruit program staff for staff surveys and interviews and program participants for focus groups, we will use the following methods:

• Program staff will be recruited for surveys and interviews based on whether they have been asked to participate in the implementation of a strategy as a part of their job responsibilities. For example, we would recruit program facilitators and their supervisors to provide insight on a workshop-based engagement strategy. Program managers and leaders overseeing all program operations will also be recruited for qualitative data collection.

• A convenience sample of program participants will be recruited to participate in focus groups based on their exposure to a strategy and their interest and willingness to participate in a focus group. For example, if the program is implementing a workshop-based engagement strategy, we would ask the program for help identifying a group of program participants who attended workshop sessions in which the strategy was deployed.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The instruments were adapted from similar RCL protocols and surveys used for other Mathematica projects, including ACF-funded projects to examine educator training for youth-serving HMRE grantees and to strengthen the implementation of two-generation programs serving parents and children in the same family.¹ These instruments from other projects were flexible enough to account for the range of question types necessary to gather information in learning cycles across different programs, but specific enough to provide useful information. Therefore, we believe the adapted instruments will be similarly effective for SIMR.

The instruments are purposefully broad to allow for flexibility across programs and respondents. The participant focus group protocol (Attachment B), for example, covers a range of topics related to recruitment, program participation, case management, and staff-participant relationship. The staff interview topic guide (Attachment C) and staff survey (Attachment A) outline topics related to use of a strategy, perceived effectiveness of the strategy, data collection, and suggestions for improvement. When conducting a rapid learning cycle, the study team will employ only questions on the survey or qualitative tools that are useful in responding to the key learning objectives of the cycle.

These data collection instruments will not be pretested. Previous RCL projects have successfully used similar instruments.

B4. Collection of Data and Quality Control

Three study team members (one lead and two supports) will be assigned to each grantee to support RCL. Each team of three will conduct or oversee surveys and qualitative data collection with staff and participants.

All senior team members leading the RCL teams have experience conducting surveys and qualitative data collection to support RCL and have experience working with community-based organizations. All team members will receive training to ensure that grantees are engaged in a consistent manner. With participant permission, virtual interviews and focus groups will be recorded, and a junior staff will take notes during all interviews and focus groups. Lastly, to ensure quality and consistency, the team will meet frequently to discuss program activities and troubleshoot issues as they arise.

No data evaluation activities are planned.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The surveys and qualitative data collection activities are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be reported.

Nonresponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Data collected for RCL will support internal decision-making at individual programs and build evidence for promising practices related to recruitment, retention and engagement in HMRE programming. The data will not be used to generate population estimates for dissemination.

B7. Data Handling and Analysis

Data Handling

No personally-identifiable information will be given to anyone outside of the SIMR study team and ACF. Typed notes and audio recordings will be stored on Mathematica’s network, which is accessible only to the SIMR study team, and destroyed upon completion of the study.

Data Analysis

This project will not employ complex analytical techniques on the instruments included in this information collection request. To analyze qualitative data, such as notes from staff interviews and participant focus groups, we will use standard qualitative analysis techniques such as thematic identification. For the staff survey, we will conduct standard qualitative analysis of responses to open-ended items and calculate ranges averages, and simple descriptive statistics for the remaining questions.

Data Use

The study team will release a final report describing knowledge gained about common implementation challenges and solutions. The report will provide details about the design and general insights, lessons, and themes from conducting rapid cycle learning with HMRE programs to aid with interpretation. When necessary, results will be labeled as examples. The products will not share quantitative findings about the programs. In sharing findings, we will describe the study methods and limitations to generalizability and as a basis for policy.

B8. Contact Persons

• Robert Wood, Mathematica, project director, [email protected]

• Samantha Illangasekare, OPRE, contracting officer’s representative, [email protected]

• Shirley Adelstein, OPRE, contracting officer’s representative, [email protected]

Attachments

Attachment A, SIMR Staff Survey

Attachment B, SIMR Participant Focus Group Protocol

Attachment C, SIMR Staff Interview Topic Guide

1 Self-regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM, OMB Clearance #0970-0355, approved June 2018) and Next Steps for Rigorous Research on Two-Generation Approaches (OMB Clearance #0970-0356, approved July 2020).

7