Supporting Statement A - GenIC Formative for Program Support - SIMR

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Strengthening the Implementation of Marriage and Relationship Programs (SIMR)

Formative Data Collections for Program Support

0970-0531

Supporting Statement

Part A

March 2021

Submitted by:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4th Floor, Mary E. Switzer Building

330 C St., SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

Shirley Adelstein

Executive Summary

• Type of request: This information collection request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of request: As an initial phase for the Strengthening the Implementation of Marriage and Relationship Programs (SIMR), the project team gathered preliminary information about potential program sites as approved under the umbrella generic: Formative Data Collections for ACF Research (0970-0356; approved October 26, 2020). For this phase of the project, we are seeking clearance to conduct rapid-cycle learning (RCL) activities (participant focus groups, staff interviews, and staff survey) to pilot and refine promising solutions with healthy marriage and relationship education (HMRE) grantees in the SIMR study. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time sensitivity: Grantees in SIMR will begin providing HMRE services on April 1, 2021. In order to begin the rapid-cycle learning process when grantees begin providing services, we request approval by March 31, 2021.

Part A

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) proposes to conduct rapid-cycle learning (RCL) activities with grantees funded by the Office of Family Assistance (OFA) for the purpose of piloting and refining promising strategies to address implementation challenges frequently faced by healthy marriage and relationship education (HMRE) grantees.

Since 2005, Congress has authorized dedicated funding for discretionary grants from the OFA to promote HMRE programs. ACF’s Office of Planning, Research, and Evaluation (OPRE) launched the Strengthening the Implementation of Marriage and Relationship Programs (SIMR) study to understand implementation challenges faced by HMRE programs and test strategies to address these challenges. The RCL activities proposed through this generic information collection (GenIC) will allow the study team to engage with HMRE programs in a highly collaborative and individualized process to develop tailored strategies to address implementation challenges in their programs, test the strategies, and build evidence for the field that helps HMRE programs address implementation challenges and improve their implementation. The ultimate goal is to strengthen programs’ effectiveness and evaluation capacity.

ACF contracted with Mathematica and Public Strategies to implement the SIMR project. Participating grantees are those awarded five-year grants in 2020 and selected for inclusion in the SIMR study.

Legal or Administrative Requirements that Necessitate the Collection

This is a discretionary data collection authorized under Sec. 811 (b) Healthy Marriage Promotion and Promoting Responsible Fatherhood Grants of the Claims Resolution Act of 2010, Pub. L. No. 111-291, 124 Stat. 3064 (Dec. 8, 2010).

A2. Purpose

Purpose and Use

The purpose of this information request is to test and refine strategies to address implementation challenges that HMRE programs commonly face. This proposed information collection meets the following goal of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Use of rapid-cycle testing activities to strengthen programs in preparation for summative evaluation.

This proposed GenIC includes staff surveys, participant focus groups, and staff interviews as part of RCL activities in selected HMRE program sites. The goal of the SIMR RCL activities is to generate evidence about how to strengthen HMRE programs by testing and refining strategies to overcome common implementation challenges related to recruitment, retention, and engagement. The lessons learned through these cycles will provide guidance to other HMRE programs on how to address implementation challenges that they may face in their programs.

The study team will partner with grantees to implement solutions tailored to the context and specific challenges of each grantee and conduct multiple learning cycle assessments of the solutions. Early cycles will seek to understand the feasibility of implementing a strategy, while later cycles will examine the success of the solutions through more rigorous methods.

The intended use of the information collected through this current request is to provide technical assistance to HMRE programs to help them address common implementation challenges and build evaluation capacity. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information. The remainder of this section describes the study team’s plans for conducting RCL activities and how the information will be used.

Study Objectives

The SIMR RCL activities have three key objectives:

• To strengthen the participating programs by iteratively refining the implementation strategy that the program targets.

• To document lessons learned about the strategies being tested to inform the broader field.

• To prepare programs for a potential summative evaluation of the overall effectiveness of their program conducted through either a local or federal evaluation.

Study Design

Through this data collection effort, we will work with 10 HMRE grantees to iteratively test and refine strategies through RCL. Under the formative generic clearance for research and evaluation (0970-0356, approved October 2020), the study team first gathered information about program plans, experiences, and the feasibility of implementing specific solutions to common implementation challenges and engaging in the rapid-learning activities. These findings informed the study team’s assessment of the most relevant implementation challenges, feasible solutions, and programs that may benefit from RCL activities. Under this clearance, the study team also selected the 10 sites to engage in RCL activities (for information about participating sites, see Supporting Statement B).

Starting in April, selected sites will conduct short, iterative tests of the strategies. These rapid learning cycles are designed to assess the feasibility of implementing the strategy, refine implementation of the strategy in response to feedback, and ultimately, provide suggestive evidence of the success of the strategy at addressing the specified implementation challenge. We expect that each site will complete up to five iterative cycles. Learning cycles will begin in April 2021, pending OMB approval, and will continue over a period of up to 18 months.

During a cycle, a set of program staff (such as recruiters, case managers, or program facilitators, depending on the focus of the strategy) will implement the strategy with a group of program participants. Each cycle may last between 2 and 12 weeks. During the cycle, program staff will provide contemporaneous feedback on their knowledge, comfort, self-report of implementation fidelity, perceived success of the strategy, and suggestions for improvement through short web surveys (Attachment A). These surveys will be administered repeatedly (up to once per week, depending on the program calendar and frequency of strategy use) to capture changes in responses over time. For example, the staff survey will allow the study team to understand whether staff members’ comfort with a strategy increases the longer that they implement it.

In addition, program staff will be asked to participate in semi-structured interviews (Attachment C) to provide overall reflections on their use of the strategy during the rapid learning cycle. Interviews aim to gather perspectives from a range of staff in the HMRE program, including program directors, managers, supervisors, and frontline staff. These interviews will be conducted near the end of each learning cycle.

Also near the end of each learning cycle, the study team will gather feedback from program participants through interviews or focus groups (Attachment B). This data collection is designed to understand participants’ program experiences and overall satisfaction with the services they have received and the staff with whom they have interacted. Where possible, the study team will use data from the ACF-sponsored Information, Family Outcomes, Reporting, and Management (nFORM) system on relevant outputs (such as number of enrollments or workshop attendance rates) and outcomes (such as participant satisfaction responses on exit surveys) to provide suggestive evidence of the success of the strategy. The nFORM information collection request is currently under review by OMB.

After a cycle is concluded, the study team will organize findings into themes and give a presentation to key site staff members. The goal of the presentation will be for site staff members to reflect on the themes, identify additional lessons, and identify opportunities for additional improvement. Together, the study team and site staff members will refine the strategy and plan for the next iterative learning cycle.

Universe of Data Collection Efforts

The current request includes three main data collection efforts (summarized in the table below). All data collection instruments will be tailored and shortened for each individual site and the strategy that it is testing.

1. SIMR Staff Survey (Attachment A): This survey will be administered to frontline program staff on either a weekly or bi-weekly schedule, depending on the needs of the specific learning cycle, to gather information about the implementation strategy being tested. The study team expects to administer staff surveys bi-weekly during 10 of the 18 months and weekly during 8 of the 18 months. The study team will tailor the survey to each site to include only the subset of questions relevant to the program and implementation strategy being tested.

2. SIMR Participant Focus Group Protocol (Attachment B): The study team will conduct up to seven participant focus groups per site. Each focus group will include up to 8 participants and will be conducted using a virtual video-conferencing platform. Given that the COVID-19 pandemic has made it difficult to safely convene a group of people and that program participants are likely to be facing a number of barriers to participation in a virtual focus group (including schedule challenges and technological limitations), we have designed the protocol so that it can be adapted for use with individual participants as a semi-structured interview protocol. The protocol will be tailored based on the strategy that is the focus of rapid-cycle learning.

3. SIMR Staff Interview Topic Guide (Attachment C): The study team will conduct semi-structured qualitative interviews with up to 10 program staff in each of the sites in each of the 7 learning cycles. We will aim to include 1 program director, 1 program manager, 2 supervisors, and 6 frontline staff. The interview guide shows the coverage of topics for each of these respondent types. It will be tailored based on the strategy that is the focus of rapid-cycle learning and for the specific staff roles within each site (for example, in some sites, the program manager may also be responsible for supervising frontline staff). The study team will submit for review a nonsubstantive change request for the staff interview topic guide showing how topics in the guide are worded as questions for semi-structured interviews.

Data collection activity	Instruments	Respondent, content, purpose of collection	Mode and duration
Staff Survey	SIMR Staff Survey	Respondents: Frontline staff Content: Staff’s use of strategy being tested, details of use of strategy, and staff perceptions on strategy. Purpose: To capture whether the strategy is being utilized, how it’s being utilized, and staff perceptions and experiences with the strategy.	Mode: Web Duration: 10 minutes per response (estimated 52 responses per respondent)
Participant Focus Group	SIMR Participant Focus Group Protocol	Respondents: Program participants Content: Experiences with recruitment and enrollment, use of services, case management, relationship with staff and other participants, and overall satisfaction. Purpose: To gauge participant perceptions and determine whether further revisions could improve the strategy.	Mode: Video conference or in-person Duration: 60 min for interviews or up to 90 minutes for focus groups
Staff interview	SIMR Staff Interview Topic Guide	Respondents: Program leaders and managers, program supervisors, and frontline staff Content: Feedback on training, guidance, and materials; use of strategy; participant responsiveness; effectiveness of strategy; suggestions for improvement; and partner organization involvement Purpose: To capture whether strategy is being utilized, how it’s being utilized, and staff perceptions and experiences with the strategy.	Mode: Video conference or in-person Duration: 45 minutes

Other Data Sources and Uses of Information

The study team plans to use the information collected from the three data collection efforts described above in conjunction with nFORM data. As a condition of their grant, HMRE grantees are required to enter data into nFORM about participant and process outcomes such as information gathered at participant intake, workshop attendance and retention, and participant responses to pre- and post-program surveys about changes in attitudes, behaviors, and skill acquisition. The nFORM information collection request is currently under review by OMB.

As applicable, the study team may request that study sites share relevant program existing documents, such as recruitment analytics or curriculum fidelity checklists that they complete as part of their existing program practices.

A3. Use of Information Technology to Reduce Burden

The study team plans to use improved information technology wherever possible. In response to the COVID-19 pandemic, the SIMR study team plans to conduct focus groups and interviews by video conference. This virtual format should be less burdensome to grantee staff, since they do not have to host study team members for in-person activities. The study team may conduct on-site, in-person data collection if circumstances change and it becomes safe to do so. Regardless of circumstances, the staff survey will be available to staff as an online survey. We will provide a link via email that program staff can use to access and complete the survey using a tablet, smartphone, or laptop.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The SIMR study team is not collecting information that is available elsewhere. None of the instruments ask for information that can be reliably obtained through other sources.

A5. Impact on Small Businesses

We expect most of the programs in the study will be small, non-profit organizations. The SIMR study team will only request information required for the intended use. The burden for respondents will be minimized by restricting the interview and survey length to the minimum required, by conducting interviews at times convenient for the respondents, and by not requiring record-keeping on the part of the programs.

A6. Consequences of Less Frequent Data Collection

A key goal of the study is to be able to refine and strengthen programs through iterative testing, which necessitates repeat data collection. Without repeat data collection, it would be difficult to assess the feasibility and effectiveness of the strategies that are implemented. The approach attempts to limit the scope of data collection to just the information needed to assess the success of the strategy being implemented.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 13, 2020, Volume 85, Number 198, page 64480, and provided a sixty-day period for public comment. ACF did not receive any substantive comments.

Consultation with Experts Outside of the Study

Several experts in HMRE programming and research provided consultation to the study team and ACF on multiple occasions throughout 2020. These experts have helped identify common implementation challenges facing HMRE programs and the solutions that may address these challenges. Experts also provided input on priority solutions we will test as part of SIMR.

A9. Tokens of Appreciation

Tokens of appreciation of $35 for this data collection effort are only planned for the participant focus groups. Focus group data are not intended to be representative of the experiences of all participant experiences in HMRE programs. However, it is important to recruit participants with a range of background characteristics to capture a range of possible program experiences. As many HMRE programs intend to serve populations experiencing disadvantage and economic hardship, including single parents, the focus group participants are likely to have low incomes. Without offsetting the direct costs of participating in the focus groups, such as arranging child care, the research team increases the risk that only individuals able to overcome financial barriers to attend will participate in the study.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally identifiable information

This data collection effort will collect personally identifiable information (PII) from program staff (names, work email addresses and telephone numbers) and program participants (names, phone numbers, and email addresses) for the purposes of arranging data collection (including scheduling and sending invitations to virtual data collection activities) and sending tokens of appreciation (if digital gift cards are used). Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of privacy

Information collected will be kept private to the extent permitted by law. Issues of privacy will be discussed during training sessions with staff working on the project. All Mathematica staff complete online security awareness training when they are hired and receive annual refresher training thereafter. Training topics include the security policies and procedures found in the Mathematica Corporate Security Manual. Program staff will transfer records to Mathematica using a secure file transfer protocol site, in case the files contain personally identifiable information.

Program participants invited to participate in a focus group or interview will be assured that information requested from them is for program improvement purposes only and that their identities will not be disclosed to anyone outside the study team. With participant permission, focus groups or interviews will be recorded, and participants will be assured that their recorded comments will be saved only until transcribed, and specify that the transcription summaries will not reveal their identities.

Data security and monitoring

No information will be given to anyone outside of the SIMR study team and ACF. All PII, typed notes, and audio recordings will be stored on restricted, encrypted folders on Mathematica’s network, which is accessible only to the study team.

A11. Sensitive Information

There are no sensitive questions as part of the staff interview guide or survey.

The participant focus group guide includes questions about reasons for enrolling in the program, relationships with staff and peers, and challenges with participating in the program, which some program participants might consider sensitive. However, these questions are essential to capture the effectiveness of the strategy being tested. The SIMR study team will obtain active consent in all sites and will inform potential study participants of the purpose of the data collection and that they may decline to answer any question.

A12. Burden

Explanation of Burden Estimates

• Staff survey (Attachment A): The study team will administer a 10-minute web survey to up to 6 frontline program staff members in each of the 10 sites, for a total of 60 respondents. The team will administer these surveys with staff multiple times over the 18-month period. The frequency will be either weekly or bi-weekly, depending on the needs of the specific learning cycle. The study team assumes they will administer staff surveys bi-weekly during 10 of the 18 months and weekly during 8 of the 18 months for a total of 52 surveys per frontline program staff member ([10 months * 2 surveys/month] + [8 months * 4 surveys/month]).

• Focus group protocol (Attachment B): Program participants will participate in focus groups. There will be up to five focus groups in each of the 10 sites, each with up to 8 participants, for a total of 560 observations (7 focus groups * 10 sites * 8 participants)

• Topic guide (Attachment C): Up to 10 program staff in each of the 10 sites (100 staff, total) will participate in semi-structured interviews in each learning cycle, for a total of 700 observations (10 staff * 10 sites * 7 cycles). The 60 staff who are given the staff survey will also be asked to participate in interviews.

The instruments as written and submitted deliberately include more topics/questions than there is time for in the length allocated to the data collection activities in the burden table. Individual study team members working with the sites will select the questions most relevant to what their sites are working on and drop irrelevant questions. It is anticipated that all questions and topics included in the instruments submitted as a part of this information request will be asked in at least one of the participating sites.

Estimated Cost to Respondents

To compute the total estimated cost, the total burden hours were multiplied by the estimated average hourly wage for program staff and participants (see table above). According to the Bureau of Labor Statistics’ Current Population Survey 2020, the median hourly wage for full-time social and community service managers is $35.50 (program managers, leaders, and supervisors) and $26.30 for community and social service specialists (frontline staff). The hourly wage ($7.25) for participants represents the federal minimum wage.

Instrument	Respondent	Total number of respondents	Number of responses per respondent	Average burden hours per response	Burden hours	Average hourly wage	Total cost
SIMR staff survey	Frontline staff	60	52	0.17	530	$26.30	$13,939.00
SIMR participant focus groups	Program participants	560	1	1.5	840	$7.25	$6,090.00
SIMR staff interview topic guide	Program leaders, managers, and supervisors	40	7	0.75	210	$35.50	$7,455.00
	Frontline staff	60	7	0.75	315	$26.30	$8,284.50
Estimated burden total					1,895		$35,768.50

A13. Costs

There are no additional costs to respondents.

A14. Estimate Costs to the Federal Government

The total estimated cost for the federal government for the data collection activities under this current request will be $1,410,968.52. This includes personnel effort plus other direct and indirect costs.

Cost category	Estimated costs
Instrument development and OMB clearance	$148,212.02
Data Collection	$1,262,756.50
Total costs over the request period	$1,410,968.52

A15. Reasons for Changes in Burden

This is an individual information collection request under generic clearance 0970-0356.

A16. Timeline

The information collected under this request will be used to test the success of various implementation strategies implemented at selected programs for the SIMR project. From January to March of 2021, the study team will identify implementation challenges and strategies and prepare for the implementation (covered under the previous clearance request, approved in October 2020). Beginning April 1, 2021, pending OMB approval of the GenIC request, the study team will work with selected programs to implement a collaboratively identified strategy and begin data collection activities. Although the primary purpose of the data collection is not for publication, the study team will develop a final report and special topics reports that share information describing work with the sites and lessons learned about how to address common implementation challenges for HMRE programs. The main audience for these publications will vary, but will include HMRE practitioners, researchers, advocates, and other stakeholders in the broader field of family strengthening programming.

A17. Exceptions

All instruments will display the expiration date for OMB approval. No exceptions are necessary for this information collection.

Attachments

Attachment A, SIMR Staff Survey

Attachment B, SIMR Participant Focus Group Protocol

Attachment C, SIMR Staff Interview Topic Guide