Supporting Statement B - Strengthening the Implementation of Responsible Fatherhood Programs (SIRF)

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Responsible Fatherhood Programs (SIRF)

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

April 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Katie Pahigiannis

Kriti Jain

Part B

B1. Objectives

Study Objectives

The Strengthening the Implementation of Responsible Fatherhood Programs (SIRF) study will use iterative learning methods to test promising practices for addressing critical implementation challenges in Responsible Fatherhood programs. To achieve this goal, the SIRF team identified common implementation challenges and potential solutions, have identified programs to undertake iterative learning activities, and work with sites on iterative learning activities to strengthen implementation and increase capacity for participating in summative evaluations. The results from this project are intended to inform future large-scale impact evaluations of programs that adopt the promising practices. The current request builds on two previous information collections under the Formative Data Collections for Research and Evaluation (0970 – 0356), which collected information on common implementation challenges faced by fatherhood programs and how promising solutions to those challenges could be tested using rapid cycle methods in program settings.

The information learned from conducting these activities will allow the team to understand how the interventions are implemented during the rapid cycles. It will allow the SIRF study team to assess how well the interventions were implemented and whether there were any implementation problems. This will also inform the SIRF study team’s technical assistance to the SIRF programs to strengthen their interventions. This information will be critical in helping the SIRF study team decide what to test in the next round of rapid cycle testing. It will also be critical in helping the SIRF study team understand the results of the tests – specifically helping the SIRF study team better understand the intervention that produced the results. This information collection request will also enable the project to include one fatherhood program that is not a Responsible Fatherhood grantee by approving that program to enter baseline characteristics, enrollment, and participation data into nFORM (Information, Family Outcomes, Reporting, and Management), a performance measures data collection system designed for Responsible Fatherhood grantees.¹ This will allow the project to learn from a broader set of fatherhood programs.

Generalizability of Results

This study is intended to present a description of fatherhood programs, the implementation challenges they face, and how they implemented innovative practices to address those challenges in participating sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

Fatherhood programs face challenges recruiting fathers, enrolling them in services, and keeping them actively engaged in services, which in turn makes obtaining rigorous evidence on program effectiveness difficult. To address these challenges, the SIRF project will use an iterative learning method (that is, rapid cycle evaluation) to identify and test promising practices to address critical implementation challenges in fatherhood programs. Analytical methods such as interrupted time series, comparison group designs, and other appropriate methods will be used to test the strategies being implemented by responsible fatherhood programs. The study outcomes will be measured using existing data from nFORM, an existing data collection required of all Responsible Fatherhood grantees (Healthy Marriage and Responsible Fatherhood Performance Measures and Additional Data Collection; ICR Ref #202102-0970-014). While nine of the ten SIRF sites are grantees, we are requesting additional burden to cover collection of data on applicant characteristics, service delivery, and program operations through nFORM for one non-grantee site.

See Supporting Statement A, section A2 for the study research questions and detail about the study design. Each instrument is designed to collect the information necessary to answer the key study questions efficiently and to maximize knowledge gained without causing undue burden.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

Instruments 1, 2, and 4 will be completed by staff and Instruments 3 and 5 will be completed by fathers at ten fatherhood programs. Staff are from organizations representing a range of programs across geographies and target populations such as young fathers, fathers with criminal-justice involvement, and those who speak English as a second language. Fathers (the program participants) are defined as current recipients of services at the selected organizations.

The study team will engage up to 10 staff and up to 135 fathers at each program in various data collection activities; the final numbers will vary by program and intervention being tested. The target population for Instrument 6 is all fathers applying to a non-grantee fatherhood program during the course of the SIRF study (estimated to be up to 500 respondents). The target population for Instrument 7 is three staff members who would be responsible for entering data into the nFORM performance measures data collection system at the non-grantee program. The target population for Instrument 8 is the site administrator.

Sampling

To ensure that SIRF findings reflect the broad set of individuals involved in SIRF, the study team will seek input from program managers, frontline staff, and fathers; service delivery partners will also be included where applicable. Up to six program managers, frontline staff, and service delivery partners will be selected to participate in semi-structured discussions (Instrument #4) at each site based on their roles related to the iterative learning cycles and interventions being tested. The SIRF team’s experience with fatherhood programs suggests that most programs will not have a large enough number of staff to sample from so all staff meeting the criteria are likely to be included. The same approach will be followed for staff reflections of each learning cycle (Instrument #2). In contrast, we will use convenience sampling to select up to 10 fathers at each site to be invited to participate in semi-structured discussions (Instrument #5). This sample will include fathers identified by program staff as likely being willing to participate or fathers who volunteer to participate in the semi-structured discussions. Sampling will not be used for SIRF Observation (Instrument 1); Learning Cycle Managers will observe staff involved in the coaching intervention once a week and all staff will provide self-reflections once a week, regardless of the intervention they are involved in. Sampling will not be used in seeking reflections from fathers either (Instrument #3); program staff will be asked to reach out to all enrolled fathers to request they complete the short web-based, mobile-friendly survey.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Each data collection instrument asks respondents to answer only questions that are necessary to achieve the objectives of the data collection. The data collection instruments will not be pre-tested; previous large-scale evaluations have successfully used similar observation tools and discussion topics, such as the Building Bridges and Bonds Evaluation (OMB Control No.: 0970-0485), during the information gathering process. The instruments in this package build off successfully executed data collection instruments of the second SIRF OMB package under the formative generic clearance, particularly the initial phone outreach protocols. Furthermore, experts Cynthia Osborne (Associate Dean for Academic Strategies; Director, Center of Health and Social Policy; Director, Child and Family Research Partnership; Director, Prenatal-to-Three Policy Impact Center, University of Texas, Austin) and David Pate (Chair and Associate Professor, School of Social Work, University of Wisconsin) reviewed the instruments included in this data collection request and adjustments were made, based on their input, that reduced burden for program staff and fathers.

We are seeking approval to have one non-grantee site collect data from nFORM. These data collection instruments are a part of the Healthy Marriage and Responsible Fatherhood Performance Measures and Additional Data Collection (See ICR Ref #202102-0970-014 for information about the development of those instruments).

B4. Collection of Data and Quality Control

Contractors to ACF, led by MDRC and including researchers from MEF Associates and Insight Policy Research (collectively called “the study team”), as well as designated staff at each SIRF site will gather information described within this package. The collection and quality control of each instrument are described below.

The Learning Cycle Manager, who is the primary liaison with SIRF study team and coordinator of learning cycle implementation, at each SIRF site will conduct observations of the iterative learning intervention (Instrument #1); line staff will also be requested to fill out an observation form once a week as a self-reflection. Learning Cycle Managers and staff will be trained on how to use the observation instrument. Staff implementing a coaching intervention will be observed by the Learning Cycle Manager once per week; all line staff, regardless of intervention, will be asked to fill out an observation as a self-reflection once a week. Completed observation instruments will be shared with the SIRF study team at the end of each learning cycle and reviewed for quality and consistency; questions about the contents will be clarified with each Learning Cycle Manager.

The Learning Cycle Manager and select front line staff at each program will be asked to fill out a Qualtrics web-based and mobile-friendly survey (reflection) toward the end of each iterative learning cycle (Instrument #2). The responses will be reviewed by the SIRF study team to prepare for discussions with each site about how to adjust the intervention approach for the next learning cycle. Likewise, the SIRF study team will seek input from fathers about their experiences with each fatherhood program in the SIRF study (Instrument #3). The team will use Qualtrics to set up a short web-based and mobile-friendly survey. Since SIRF will not have personally identifiable information about the fathers (and therefore will not have contact information), the team will ask staff to invite fathers in each cycle to fill out the survey by sharing links via email, text message, or scannable QR codes. Responses from both the staff and father reflections – which will not include any personally identifiable information - will be saved on MDRC’s secure network and will be shared with their respective programs as part of the reflection process for the learning cycle.

Toward the end of the SIRF study period, the SIRF study team will hold one-time, semi-structured discussions with program front line staff and managers (Instrument #4). Interviewers from the SIRF study team – who have experience with qualitative data collection methods – will be trained to use the instrument and how to record information they gather. Site staff will be identified by the SIRF study team, in conjunction with the Learning Cycle Manager, based on who has been involved in the interventions and who is best suited to answer the questions of interest. Notes from each discussion will be reviewed by a study team member; interviewers may be asked to revise their notes for clarification purposes.

Also, toward the end of the SIRF study period, the SIRF study team will ask to hold semi-structured discussions with fathers (Instrument #5). Interviewers from the SIRF study team – who have experience with qualitative data collection methods – will be trained to use the instrument and how to record information they gather. The recruitment protocol may vary by site – depending on their size and how fathers interact with the program. Notes from each discussion will be reviewed by a study team member; interviewers may be asked to revise their notes for clarification purposes.

We will ask the non-grantee SIRF site to enter data into the nFORM performance measures data collection system operated by Mathematica. Mathematica will be providing the SIRF study team with accounts to allow them to download de-identified data for all SIRF sites. The SIRF team will check in with the non-grantee on a regular basis to make sure they are entering participation data regularly into nFORM (Instrument #7). All program applicants will be required to fill out the Applicant Characteristics survey as part of the intake process (Instrument #6). Site administrators will also be asked to complete the Program Operations Survey (Instrument #8).

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The data collection activities in this request are not designed to produce statistically generalizable findings but rather for the purpose of building an understanding of what interventions may help to address critical implementation challenges in fatherhood programs. As with program staff, data collection for fathers (Instrument #3 and #5), participation is wholly at the respondent’s discretion. Nonetheless, response rates will be calculated and reported. Semi-structured discussions with fathers will be scheduled, based on input from program staff, to minimize burden of participating. Fathers will be invited to participate through multiple modes of communications from the study team through the program staff, such as a recruitment flyer or email text that is provided to sites to share with fathers (Appendix B). Small tokens of appreciation - $20 gift cards – will be offered to fathers participating in semi-structured discussions to offset incidental costs that may interfere with participation, such as transportation and/or child care. Note, the SIRF study team will not have any contact information for fathers and therefore cannot directly communicate with them. The value of fathers contributing to our understanding of fatherhood programming will be emphasized in all communication.

Examples include observations (Instrument 1) and written documentation (Instrument 2). Staff member participation in semi-structured discussions are voluntary. To encourage participation, staff will be fully informed of the purpose the discussions and will be scheduled to best accommodate staff schedules and minimize disruption to their workdays.

The non-grantee SIRF site will be asked to use nFORM to collect applicant characteristics data from fathers at the time of intake (Instrument #6), information on service delivery throughout the study (Instrument #7), and provide program operations information (Instrument #8). The study team does not anticipate calculating response rates for these activities. For the Applicant Characteristics survey (Instrument #6), the team expects that all fathers entering the program during the study will complete this since it will be a standard part of the intake process. The team also expects a non-grantee site to enter fairly complete information about the father and his participation into nFORM (Instrument #7) since part of the site payment will be designated to cover staff time entering data into nFORM and because the team will be monitoring nFORM data quality on an ongoing basis and providing regular feedback on the data quality. The team also expects complete information about program operations quarterly (Instrument #8).

NonResponse

As participants in the data collection described here will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.

The study team’s experience with similarly designed data collection approaches has been good with nearly all staff ever asked to participate in data collection activities agreeing. The SIRF team will emphasize the potential benefits of participating in data collection activities to increase motivation to participate; these benefits are not to the individual but to the fatherhood field broadly. To encourage fathers to provide insights about their experiences with the fatherhood programs (Instrument #3) the study team will provide each program with clear information about the data collection effort for staff to relay to fathers (Appendix A). The SIRF study team will also provide programs with written materials, like a postcard, they can provide to fathers to inform them of upcoming opportunities to provide their input (Appendix B). We will rely on the support of the Learning Cycle Manager at each organization to support our requests for scheduling meetings with staff and fathers. The SIRF team will use the e-mail template in Appendix C as a script to outreach for discussions with staff and Appendix D will be shared with staff to use as a script to outreach for discussions with fathers. The SIRF study team will not have contact information for fathers so program staff will be the conduit for the requests.

B6. Production of Estimates and Projections

Estimates will not be produced for Instruments 1 through 5 of this data collection.

The team will use the service delivery data from the one non-grantee site that the staff person enters into nFORM (Instrument #6) to calculate measures of enrollment, participation, and retention in services, and to estimate the effect of the intervention on these measures.

B7. Data Handling and Analysis

Data Handling

The Learning Cycle Manager will send all completed observation instruments (Instrument #1) to the SIRF study team at the end of each learning cycle through Box. It will be reviewed for quality and consistency; questions about the contents will be clarified with each Learning Cycle Manager.

The Learning Cycle Manager and select front line staff at each program will be asked to submit written documentation reflecting on how the intervention went in the learning cycle (Instrument #2). This will be submitted using Qualtrics; responses to reflection questions will automatically be shared through the SIRF study team’s secure network. The information provided in each submission will be reviewed by the SIRF study team and questions about the content may be clarified in phone calls with each site to the extent possible. The form will have built in checks to ensure it is filled out completely. Fathers will also be asked to reflect on their participation in the fatherhood program through the same Qualtrics system (Instrument #3). The team will use Qualtrics to set up a short web-based and mobile-friendly survey, with built in checks to ensure it is filled out completely. Since SIRF will not have personally identifiable information about the fathers (and therefore will not have contact information), the team will ask staff to invite fathers in each cycle to fill out the survey by sharing links via email, text message, or scannable QR codes. Responses – which will not include any personally identifiable information - will be saved on MDRC’s secure network and will be shared with their respective programs as part of the reflection process for the learning cycle.

Key points of each discussions with program staff and fathers (Instruments #4 and #5) will be documented in Microsoft Word by a note-taker who will be present during each discussion. The information gathered will be kept private and protected securely. The study team is required to use encrypted laptops, secure storage locations, secure transfer mechanisms (when necessary), and access to secure locations will only be granted to those on a need-to-know basis.

The note-taker will have the discussion lead review the notes to ensure accuracy.

Staff from the non-grantee will enter demographic information about program applicants and information to track participant engagement in the program into nFORM (Instrument #6 and #7). The study team will monitor data entry into nFORM for the non-grantee site by checking the data regularly and discussing the enrollment, participation, and retention numbers with program staff to make sure it is consistent with their expectations. Like many other fatherhood programs, the non-grantee may also collect more comprehensive data in their own MIS. Therefore, the non-grantee will know what the enrollment, participation, and retention numbers should look like for their program and know if nFORM data entry is accurate.

Data Analysis

In the immediate term, written documentation gathered from staff near the end of each learning cycle (Instrument #2) will be compiled by site for the SIRF study team to use as background for reflecting on the learning cycle and preparing for the next one. In the longer term, all written documentation will be analyzed together across learning cycles and sites to identify cross-cutting themes and lessons.

Also, in the immediate term, responses from fathers (Instrument #3) will be shared with each site near the end of each learning cycle; this information will be used as part of the learning cycle reflection process. In the longer term, all father responses will be analyzed together across learning cycles and sites to identify cross cutting themes.

The notes from semi-structured discussions with program staff (Instrument #4) and fathers (Instrument #5) will be coded using MDRC’s qualitative analysis software. The coding structure will be organized around primary research questions identified a priori. We will also code for emergent themes and further develop the coding structure, analyzing patterns of consensus and divergence in themes particularly across sites targeting similar iterative learning outcomes.

The SIRF team and non-grantee staff will have access to data entered into nFORM in two ways: 1) individual level data through the data export and individual level reports, and 2) aggregate level data through operational reports and data visualization tools. nFORM data will be used to assess the impact of the interventions on program enrollment, participation, and retention. The analytical methods employed will be appropriate to the interventions being tested and the feasibility of implementation (based on conversations with sites), but could include random assignment, interrupted time series, single case designs, comparison groups, benchmarks, or qualitative assessments. Program Operations Survey data (Instrument #8) will be analyzed, as a supplement to other data collected from program staff (such as through Instrument #4) to understand the context within which SIRF interventions operated.

Data Use

The information gathered will enable the project to determine how the SIRF interventions are being implemented within the iterative learning environment, and to estimate whether the interventions are effective at increasing enrollment, participation, and retention in services. It will allow the SIRF study team to assess how well the interventions were implemented and whether there were any implementation problems. This will also inform the SIRF study team’s technical assistance to the SIRF programs to strengthen their interventions. Findings from the current information collection request will also be incorporated into documents or presentations that are made public, such as through conference presentations, website, or social media. These dissemination products may include infographics, short briefs or reports, or how-to guides and will describe how to properly interpret, analyze, and evaluate findings from the data collection, including limitations with regard to generalizability and as a basis for policy.

B8. Contact Person(s)

Primary Project Contacts

Charles Michalopoulos, MDRC

SIRF Project Director

(510) 844 – 2235

[email protected]

Dina Israel, MDRC

SIRF Project Deputy Director

(212) 340 – 8606

[email protected]

Michelle Manno, MDRC

SIRF Information Gathering Task Lead

(212) 340 – 8873

[email protected]

Attachments

Instrument #1 SIRF Observation

Instrument #2 SIRF End of Learning Cycle Reflections for Staff

Instrument #3 SIRF Reflections from Fathers

Instrument #4 SIRF Staff Semi-Structured Discussion Topics

Instrument #5 SIRF Father Semi-Structured Discussion Topics

Instrument #6 Non-Grantee Use of nFORM’s Applicant Characteristics Survey

Instrument #7 Non-Grantee Use of nFORM – Staff Data Entry

Instrument #8 Non-Grantee Use of nFORM’s Program Operations Survey

Appendix A_About SIRF Interventions and Sites

Appendix B_Information for staff to tell fathers about opportunities to provide input

Appendix C_Postcard for fathers about opportunity to provide input

Appendix D_Outreach to program staff to schedule semi-structured interview time

Appendix E_Outreach to fathers to schedule semi-structured interview time

Appendix F_Meeting topics for program staff

Appendix G_ SIRF Project Description (has already been approved as part of package #2)

1 Data collected through nFORM and the associated burden for Responsible Fatherhood grantees has been reviewed and approved through the Healthy Marriage and Responsible Fatherhood Performance Measures and Additional Data Collection (ICR Ref #202102-0970-014)

8