SSB - Program Support GenIC: HomeEc Formative Evaluation

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Supporting Family Economic Well-Being through Home Visiting (HomeEc)

Formative Evaluation

Formative Data Collections for Program Support

0970–0531

Supporting Statement

Part B

July 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Pooja Gupta Curtin

Laura Nerenberg

Part B

B1. Objectives

Study Objectives

The Supporting Family Economic Well-Being Through Home Visiting (HomeEc) study, overseen by the Administration for Children and Families (ACF), Office of Planning, Research, and Evaluation (OPRE), aims to understand how early childhood home visiting programs have supported or could support family economic well-being. Family economic well-being—including financial, material, and socioemotional resources—can be an important support for families’ long-term stability, family functioning, and children’s healthy development. The previous study activities including a literature and document review, program and participant consultations, and information collection about how home visiting programs supported family economic well-being during the COVID-19 pandemic, has provided the study team with lessons learned that will inform the formative evaluation phase of the study. The program and participant consultations and information collection about COVID were completed under the Formative Data Collections for ACF Research Generic Clearance (Office of Management and Budget [OMB] #0970-0356). During the next stage of the project, the study team will work with select home visiting programs to co-create and test the implementation of practices aimed at supporting family economic well-being. This formative evaluation of practices will be an iterative process, enabling programs to create a defined needs statement, develop practices to address the need(s), and iteratively test and refine the practices. The related information collections are the subject of this request.

Generalizability of Results

This study seeks to present an internally valid description of the feasibility of implementing practices supporting family economic well-being in the selected program sites, not to promote statistical generalization to other programs or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The primary purpose of this information collection is to co-create practices with home visiting programs to support family economic well-being and test those practices using iterative rapid-cycle formative evaluation. The rapid-cycle formative evaluation method is appropriate for the planned uses of information because it involves learning about the unique challenges of each participating program and co-creating a practice to address that challenge, drawing on the combined expertise of researchers and program staff. The study team will then use short, iterative learning cycles for program staff to try the practice, obtain useful feedback from staff and the caregiver participants, and map next steps to refine the practice for the next round of testing. These activities aim to provide timely and useful information to participating programs about promising practices for addressing family economic well-being and strengthening programs’ capacity to support family economic well-being. They will also provide timely and useful information to ACF and HRSA as they consider potential next steps for the strategies, such as further refinement and testing or scale-up within the home visiting field. Although the primary purpose of this information collection is not for publication, the study team will summarize the findings and lessons learned across sites for how to support family economic well-being through home visiting. The main audience for such a publication will include practitioners, researchers, and other interested parties in the early childhood home visiting field. The publication will clearly state limitations, including that findings are not generalizable to the broad population of early childhood home visiting programs.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. This study does not include an impact evaluation and will not be used to assess participants’ or program outcomes. All publicly available products associated with this study will clearly describe key limitations.

B2. Methods and Design

Target Population

A general solicitation for programs to express interest was published to allow the study team to begin to identify sites that may be interested in and best suited for participation. Based on the responses to the published general solicitation about the study and potential for participation, the research team will purposively select three local home visiting programs to participate in the study. The study team will also purposively select families and program and partner staff to participate in the study. Because the study team will purposively select the programs and staff, they will not be representative of the population of early childhood home visiting programs, staff, or families.

For each of the three participating sites, the study team will collect information from a purposively selected group of staff in home visiting program sites, staff from organizations that partner with home visiting programs to support family economic well-being services, and/or caregiver participants receiving family economic well-being practices as part of their home visiting services.

• Program staff (such as program directors, supervisors, and home visitors). Program staff involved in testing the practices will participate in virtual interviews (Instruments 2 and 4) and virtual strategic planning meetings (Instrument 3) and will complete web surveys (Instrument 6).

• Partner staff. Staff from community organizations that the home visiting programs partner with to support family economic well-being services, such as an employment and training organization, may participate in virtual interviews (Instrument 2) and complete web surveys (Instrument 6).

• Caregiver participants. Caregiver participants who receive the family economic well-being practices being tested may complete web or paper surveys (Instrument 5) and a subset of the caregiver participants may participate in virtual interviews (Instrument 2).

Site Selection and Respondent Recruitment

Any home visiting program across the country can be nominated to participate in the formative evaluation. To identify program sites to recruit, the study team published a general solicitation for programs to express interest, and provide any information they feel is relevant (there is no set of questions to respond to, just a request to include contact information with whatever information is submitted). Based on information received through the general solicitation, the study team will begin conducting recruitment calls to potential sites to gauge interest and gather additional information to inform the selection process. The study team will reach out to prospective programs via email (Appendix A) to invite them to participate in a recruitment call. The study team will also share an information sheet about the study (Appendix B) with prospective programs prior to the recruitment call.

Through the outreach efforts, the study team will identify a list of three priority sites and up to nine alternate sites. In selecting the three programs to participate in the study, the study team will select sites that have shown a demonstrated interest and capacity to participate. The study team will aim to recruit programs that are diverse from one another, such as considering the home visiting model that programs implement, geography (for example, rural or urban), and population they serve (characteristics of the participating population).

When the study team has recruited the three program sites, the team will begin the process of identifying and recruiting program directors, supervisors, home visitors, partner staff, and caregiver participants using the following methods:

• The recruitment call process will enable the study team to identify a key program contact. This staff member will help select other program staff to participate in the study activities, including semistructured virtual interviews, web surveys, and strategic planning meetings.

• The key program contact working with the study team may identify partner staff who work for organizations that help home visiting programs support family economic well-being. If relevant, the study team may ask the program staff to facilitate the initial outreach and introductions so the team can invite the staff to participate in semistructured virtual interviews.

• If relevant to the practice, home visitors will recruit caregiver participants who receive the family economic well-being practices being tested to participate in the study. The study team will work with home visitors to identify and invite caregiver participants to participate in a virtual semistructured interview and/or complete a paper or electronic survey.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument

The study team developed six data collection instruments for the HomeEc formative evaluation. Table A.1 in Supporting Statement A provides details about the respondents, content, purpose, mode, and duration for each. Content experts at Mathematica developed the data collection instruments, which the contractor adapted from similar formative evaluation protocols and surveys used for other projects, including ACF-funded projects, that test and refine innovative strategies to overcome implementation challenges in other social services contexts.¹ The instrument development process was also informed by protocols from the HomeEc project’s program and participant consultations that examined the use of practices to support family economic well-being in home visiting programs.² The team adapted all questions in the recruitment and eligibility screener (Instrument 1), semistructured interview guides (Instruments 2 and 4), semistructured discussion guide for strategic planning meetings (Instrument 3), and surveys (Instruments 5 and 6) from similar formative evaluation protocols and surveys used by other studies to iteratively test and refine program practices. The instruments from previous formative evaluation studies sought to flexibly gather specific information across a range of question types necessary to understand what occurred in learning cycles in different programs testing different strategies. Therefore, the study team expects the instruments developed and adapted for this study will be similarly effective for the HomeEc project’s formative evaluation activities.

The instruments for this study are purposefully broad to allow for flexibility across programs and respondents based on the family economic well-being strategy to test. Study team staff will use only questions relevant to the respondent and the key task at hand. The study team will tailor questions within each instrument:

• The semistructured interview guide for needs identification (Instrument 2) and semistructured discussion guide for strategic planning meetings (Instrument 3) ask questions about the program’s context, challenges, and potential solutions.

• The semistructured staff interview guide (Instrument 4) and caregiver participant interview guide (Instrument 5) and the staff survey (Instrument 6) for the learning cycles ask questions about practice implementation and refinements for future testing iterations.

The study team will not pre-test these data collection instruments. Previous formative evaluation projects have successfully used similar instruments.

B4. Collection of Data and Quality Control

This information collection will undertake several data collection activities:

1. Selecting program sites. The study team will use Instrument 1 for recruitment calls to request specific information to select three programs that expressed interest in participating. During these recruitment calls, the study team will assess a program’s interest and capacity to participate, whether the program receives funding from the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program or Tribal MIECHV program, confirm the home visiting model the program uses, and learn about the partnerships in the community the program has in place. The study team will prioritize programs that receive MIECHV or Tribal MIECHV funding and seek to include programs that vary in home visiting model used, location and urbanicity, and the type of population served. The study team will also consider the family economic well-being needs programs are interested in addressing or the types of family economic well-being practices programs are most interested in using, as we may prioritize programs interested in testing similar strategies. The study team’s goal is to begin recruiting programs in August 2024, with recruitment taking place on a rolling basis until the study team, in collaboration with ACF and HRSA, selects three sites.

2. Identifying needs and practices to test through rapid-cycle formative evaluation. After selecting the programs, the study team will conduct group interviews with program staff to identify family economic well-being needs to address and brainstorm potential practices to overcome those challenges and support family economic well-being. The study team will facilitate strategic planning meetings with program staff to determine which practices to test, develop materials and training to support the selected practices, and develop implementation and testing plans. During these meetings, program staff will contribute their knowledge and expertise of their program’s unique circumstances and the caregivers they work with, and the study team will contribute information learned from available research and evidence. In between strategic planning meetings, the study team will have ad hoc individualized consultations, as needed, with program leadership to prepare for meetings, debrief findings, and discuss next steps. These consultations will involve no more than nine individuals across the three program sites. The team expects this phase to begin in November 2024 and last about seven months.

3. Testing practices through rapid-cycle formative evaluation. The study team and program staff will conduct up to three short, iterative learning cycles of six to eight weeks to test the practices, collect data on implementation, and use the data to refine the practices. During a cycle, a set of program staff, including program directors, supervisors, and home visitors, will implement the practice for four to six weeks. During that time, the study team will gather participant and program staff feedback on practice utility and implementation through surveys and interviews. Feedback will focus on program staff’s experience and comfort implementing the practice and caregiver participants’ experience receiving and satisfaction with the practice, as applicable. The study team will document the findings in a memo or slide presentation and discuss the findings and next steps with staff in each program. The goal of the presentation and associated discussion is for program staff to reflect on the findings, determine whether to continue with rapid learning, and identify additional opportunities for refinement. The study team will work closely with program staff to refine the practice. They will then test the refined practice in the next learning cycle. As needed, the study team will consult with program leadership to discuss findings and plan next steps. The team expects this phase to begin in July 2025 and be complete within about six months.

The study team will use a rapid-cycle formative evaluation design because it supports the project in identifying, testing, and refining family economic well-being practices within the home visiting context. The rapid-cycle formative evaluation involves learning about the unique challenges of each participating program and co-creating practices to address those challenges, drawing on the combined expertise of researchers, program staff, and caregivers. Short, iterative learning cycles will provide sufficient time for program staff to try out a practice, obtain useful feedback from staff and the caregiver participants, and map out next steps to refine the practice for the next round of testing. These activities should provide timely and useful information to programs about promising practices for addressing family economic well-being and strengthen programs’ capacity to support family economic well-being. Although the primary purpose of this information collection is not for publication, the study team will summarize the findings and lessons learned across sites for how to support family economic well-being through home visiting. The main audience for such a publication will include federal decisionmakers, practitioners, researchers, and other interested parties in the early childhood home visiting field. The publication will clearly state limitations, including that findings are not generalizable to the broad population of early childhood home visiting programs. Additional details on the rationale of the study design are available in Section B1 of Part B under Appropriateness of Study Design and Methods for Planned Uses.

A key limitation of the study design is that the six-month implementation and testing period means the study can only monitor outcomes for a short period. In addition, the lack of a comparison group and the study’s formative design to assess implementation and feasibility mean that evidence generated in learning cycles on the promise of practices will be suggestive and not causal. Study findings that will inform the practices will be based on a small, nonrepresentative sample of early childhood home visiting programs (three program sites). Thus, the study cannot generalize findings to the broad population of early childhood home visiting programs.

Three study team members will guide each participating home visiting program site through the formative evaluation. The site team members will conduct all virtual interviews with program staff, partner staff, and caregiver participants; facilitate all strategic planning meetings with program and partner staff; work with home visitors to disseminate and collect the caregiver participant paper or web surveys; and oversee the web survey dissemination to program and partner staff.

All senior team members leading the formative evaluation teams have experience collecting qualitative data, overseeing the administration of web and paper surveys, and facilitating virtual strategic planning meetings using human-centered design tools and activities. All members across the three site teams will receive training on study processes and procedures to ensure the study engages the three participating programs in a consistent manner. With permission from participants, the team will record all virtual semistructured interviews and a team member will take notes. Finally, to ensure quality and consistency, the senior team members and members of the three site teams will meet frequently to discuss program activities and troubleshoot issues that arise.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The data collection activities will not produce statistically generalizable findings, and participation is wholly at each respondent’s discretion. The study team will not calculate or report response rates.

NonResponse

Because the study will not randomly sample respondents, and findings are not intended to be representative, the study team will not calculate nonresponse bias. The study team will document and report respondents’ demographics in written materials associated with the data collection.

B6. Production of Estimates and Projections

Data collected for this formative evaluation will support internal decision-making at individual programs, inform refinements to the practices and their implementation, and build evidence for promising practices related to supporting family economic well-being through home visiting. The study team will not use the data to generate population estimates for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

No one outside the study team and ACF will have access to personally identifiable information. The team will store typed notes and audio recordings on Mathematica’s secure network, which only the study team can access. The study team will work with ACF to determine the data disposition and/or data archive schedule at the end of the study. The team will administer the caregiver participant survey on paper or electronically. If the survey is administered on paper, sites will return the completed paper surveys to the study team through tracked FedEx packages, which the study team will immediately scan and then save the completed surveys on a secure drive before shredding them. The HomeEc team will then data-enter responses from completed paper surveys into a secure data entry system and follow a structured process to check for data entry errors prior to analysis.

Data Analysis

This project will not employ complex analytical techniques. To analyze qualitative data, including notes from the semistructured interviews and strategic planning meetings, the study team will use standard qualitative analysis techniques such as thematic identification. For the program staff and caregiver participant surveys, the study team will conduct standard qualitative analysis of responses to open-ended items and calculate ranges, averages, and simple descriptive statistics for the closed-ended questions.

Data Use

A short public report will summarize the insights and lessons learned through the formative evaluation. The report will provide details about the design and general insights, lessons, and themes from conducting formative evaluation with home visiting programs. When necessary, the report will label results as examples. The products will not share specific survey or interview responses by program. In sharing findings, the study team will describe the study methods and limitations to generalizability and as a basis for policy.

B8. Contact Persons

The following individuals at ACF and Mathematica are involved in leading the HomeEc project:

Pooja Gupta Curtin

Social Science Research Analyst

Office of Planning, Research, and Evaluation

[email protected]

Laura Nerenberg

Senior Social Science Research Analyst

Office of Planning, Research, and Evaluation

[email protected]

Katie Eddins

Senior Researcher

Mathematica

[email protected]

Attachments

Instruments

Instrument 1: Program Recruitment and Eligibility Screener

Instrument 2: Interview Guide for Needs Identification

Instrument 3: Discussion Guide for Strategic Planning Meetings

Instrument 4: Interview Guide for Learning Cycles

Instrument 5: Caregiver Participant Survey for Learning Cycles

Instrument 6: Staff Survey for Learning Cycles

Appendices

Appendix A: Invitation to Participate in Recruitment Screener

Appendix B: Formative Evaluation Information

1 “Strengthening the Implementation of Marriage and Relationship Services.” SIMR, OMB Clearance #0970-0531, approved February 2021; “Strengthening Relationship Education and Marriage Services.” STREAMS, OMB Clearance #0970-0355, Approved September 2018; “Next Steps for Rigorous Research on Two-Generation Approaches”, OMB Clearance #0970-0356, Approved July 2020.

2 “Supporting Family Economic Well-Being through Home Visiting.” HomeEc, OMB Clearance #0970-0356, approved March 2023.

8