HomeEc SSB - Formative GenIC Research

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Supporting Family Economic Well-Being through Home Visiting (HomeEc)

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

November 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Pooja Gupta Curtin

Part B

B1. Objectives

Study Objectives

The Supporting Family Economic Well-Being through Home Visiting (HomeEc) study, overseen by the Administration for Children and Families (ACF) Office of Planning, Research, and Evaluation (OPRE), is designed to better understand, measure, and inform future research focused on improving the economic well-being of families enrolled in local early childhood home visiting (ECHV) programs. Family economic well-being—including financial, material, and socio-emotional resources—can be an important support for families’ long-term stability, family functioning, and children’s healthy development. Some ECHV models target family economic well-being and have demonstrated effects on outcomes such as parent income. Given the connections between economic stability, parenting, and caregiver well-being, a stronger focus on family economic well-being in ECHV could benefit families in many ways. HomeEc proposes to collect in-depth qualitative data to help understand the following:

• The components of family economic well-being

• Promising practices to support family economic well-being in ECHV

• How family economic well-being is measured in local ECHV programs

Generalizability of Results

The results of this study are intended to define family economic well-being, describe strategies currently being used or that could be used by local ECHV programs to support such well-being, and identify how family economic well-being could be measured in local ECHV programs. These results are intended to be relevant for ECHV programs beyond the selected sample, but because this study’s design utilizes a purposive sample and not random selection from a defined population, the results are not representative of a broader population. . . For example, the strategies described might not represent all relevant strategies to support family economic well-being currently used by local ECHV programs.

Appropriateness of Study Design and Methods for Planned Uses

The study’s purposive site selection and qualitative methods are appropriate for understanding the different perspectives of family economic well-being and approaches to improving it among different program staff and participants. Purposive site selection allows the study to include diverse local ECHV programs (different ECHV approaches) and participants (including traditionally marginalized and underrepresented groups). The data will inform other project activities, such as (1) developing and revising the conceptual model, which ACF can use to illustrate pathways for ECHV to support family economic well-being; (2) developing or adapting measures of family economic well-being for use by home visitors; and (3) conducting rapid-cycle evaluation of practices to assess their feasibility for use in ECHV programs serving different populations¹.

A purposive sample will enable us to include a range of local ECHV programs and understand the different community contexts in which such programs operate, different perspectives on family economic well-being, the ways local ECHV programs collect data on family economic well-being, and the different ways this well-being can be achieved.

Qualitative methods—such as semi-structured interviews and focus groups—will promote an in-depth examination of the different components of family economic well-being, how it is measured, and what supports local ECHV programs can provide to help families achieve short- and long-term economic well-being. The interviews and focus groups are designed to be easily adaptable to specific situations and respondent groups (more details about respondent types appear in Section B2 under Target Population).

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. This study does not include an impact evaluation and will not be used to assess participants’ outcomes. All publicly available products associated with this study will clearly describe key limitations.

B2. Methods and Design

Target Population

To achieve the study’s objectives, we will collect qualitative data from respondents at six local ECHV programs that are currently implementing family economic well-being components. For each of the six programs, the target population will include ECHV program directors, supervisors, and home visitors. The target population also will include parent participants—pregnant or parenting caregivers of a young child who receive ECHV services.

The research team will use nonprobability purposive sampling to select local ECHV programs and identify potential respondents who can provide information on the study’s key constructs. Given that participants will be purposively selected, they will not be representative of the population of ECHV programs, staff, or families.

Site Selection and Respondent Recruitment

1. Program identification

Goal: Identify a pool of programs that currently implement family economic well-being components in their ECHV programs.

The study team will use information gained through document review, literature review, and consultations with the technical working group (TWG), and federal partners to identify up to 10 potential local ECHV programs that currently implement family economic well-being components in their ECHV programs. The TWG is made up of key groups interested in the outcomes of this study, including research experts, programmatic and practice experts, research-to-practice experts, and technical assistance providers. The TWG members have direct experience with and knowledge of ECHV programs. The federal partners include Health Resources and Services Administration (HRSA) and ACF project officers who oversee the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) and Tribal MIECHV grants, as they have in-depth knowledge of the awardees and grantees and the local ECHV programs they support.

We will prioritize the recruitment and selection of local ECHV programs that are implementing clearly defined strategies to support family economic well-being, particularly if those strategies show evidence of effectiveness. These strategies could support family economic well-being in areas such as stable employment, adequate income to meet short- and long-term needs, and healthy coping with financial stress. We will also aim to maximize variation in the program characteristics, such as the racial and cultural diversity of the clients served by the programs, urbanicity, and ECHV strategies across the selected local ECHV programs. We will aim to select a mix of local ECHV programs that receive funds from MIECHV or Tribal MIECHV grants, or no MIECHV funding. The study team will make initial contact with the 10 potential local ECHV programs to introduce the study, indicate that the program was nominated because it currently implements services for supporting family economic well-being, and collect additional information about program eligibility and family economic well-being measures (using Instrument 1). We will collaborate with community referral partners (more details on the process of identifying local ECHV programs appear in Section A2 in Part A under Study Design).

2. Program selection

Goal: Identify six local ECHV programs (and four backup local ECHV programs) with the strongest potential for enabling the research team to answer the study research questions.

Using the criteria listed in Table B.1., the study team will determine which local ECHV programs are eligible for inclusion. All local ECHV programs will meet the parameters listed in the required criteria column and some or all of those listed in the preferred criteria column. The team will rank the local ECHV programs based on the number of preferred criteria met. The local ECHV programs will be further assessed on whether a program adds diversity to the sample through variation in the ECHV model, demographics of families served, geographic location, family economic well-being service format, organizational structure, and funding.

Table B1. Eligibility requirements and other considerations

Required criteria		Preferred criteria		Additional considerations
Operating at least 1 ECHV model (evidence-based or promising practice)		Specialized staff providing services for or supporting family economic well-being		ECHV model diversity
Implementing family economic well-being components in the program for at least six months		Collects information on family economic well-being from families in the program		Diversity in families served
		Uses culturally relevant screening and assessment tools		Variation in urbanicity—rural/urban; metro/non-metro
		Uses family economic well-being practices that could be tested using a rapid cycle evaluation		Variation in family economic well-being service format—individual and/or group services
		Operating at least 1 MIECHV eligible, evidence-based ECHV model		Variation in program organizational structure and size
		Program collaborates with community partners to support family economic well-being		Variation in funding used to support family economic well-being services
				Part of a larger system of care

3. Program recruitment

Goal: Recruit six local ECHV programs for the qualitative data collection.

The study team will contact the six prioritized local ECHV programs selected to be part of the study based on how the team ranked them on the eligibility criteria listed in Table B.1. The study team will secure the program’s participation in the study. As appropriate, we will ask TWG members with connections to local ECHV programs of interest to facilitate an introduction between the study team and program leadership.

4. Identification and recruitment of respondents

Goal: Identify and recruit ECHV program staff and parents enrolled in the selected programs.

Once a local ECHV program has agreed to participate in the study, the study team will work with the program director to identify up to two supervisors who would be willing to participate in a virtual 60-minute interview and up to six home visitors who would be willing to participate in a virtual 90-minute group interview. (We plan to conduct group interviews with up to three home visitors at a time.) We will also ask the program director to participate in a virtual 60-minute interview.

We will also ask all six local ECHV programs to help us identify and recruit 6 to 10 parent participants per local ECHV program to participate in a virtual 60-minute focus group (with the aim of having 4 to 6 focus group parent participants per site); we will also recruit up to 14 parent participants across up to six sites to participate in a semi-structured interview. The study team is prepared to conduct the data collection activities in a language other than English at a site if necessary and we will work across programs to recruit parent participants from different racial and ethnic groups; and with different levels of family economic well-being, family structure, and family experiences in the program so the study team can obtain a wide range of perspectives. We would also consider including parents who recently have left the program, as they could provide a unique perspective as well. We will purposively select the respondents to include a range of perspectives needed to answer the study’s research questions and inform the subsequent project tasks.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

We developed seven data collection protocols for the HomeEc study. Table A.1 in Supporting Statement A provides details about the respondent, content, purpose, mode, and duration for each. The data collection protocols will include one eligibility screener and recruitment protocol to initially screen and recruit programs (Instrument 1). They also will include four semi-structured interview protocols—one for the program director (Instrument 2), one for supervisors (Instrument 3), one for parent participants (Instrument 4), and one for home visitors (Instrument 5). Finally, they will include one semi-structured focus group guide for parent participants (Instrument 6).

All questions in the eligibility screener and recruitment protocol (Instrument 1), staff interview protocols and focus group guide (Instruments 2, 3, and 5), and interview protocol and focus group guide for parent participants (Instruments 4 and 6) are new because they reflect a new area of research, and the constructs under study cannot be measured using existing instruments. The interview protocols and focus group guides were developed by content experts at Mathematica and JBA, and informed by reviewing instruments used in similar data collection efforts, including from the Head Start REACH: Strengthening Outreach, Recruitment and Engagement Approaches with Families Study (OMB Control Number 0970-0580), the Evaluation of Child Support Cooperation Requirements in SNAP (OMB Control Number 0584-0671), the Next Generation of Enhanced Employment Strategies (OMB Control Number 0970-0356), the Family Level Assessment and State of Home Visiting (OMB Control Number 0970-0559), and the Multisite Implementation Evaluation of Tribal Home Visiting (OMB Control Number 0970-0521). Prior to data collection, we will pre-test the staff interview protocols (Instruments 2, 3, and 5) with up to three staff (program directors, supervisors, and home visitors) before beginning data collection.

The semi-structured interview protocols and focus group guide include different modules that capture the range of topics addressed in the study’s research questions. Within each module, interview questions align with the key constructs relevant to the research questions. Respondents will answer only the subset of questions that align with their own areas of experience and knowledge.

Table B.2 presents a crosswalk between the data collection instruments and the study’s objectives.

Table B.2. Crosswalk of data collection instruments and study objectives

Instrument	Components of family economic well-being	Promising practices to support family economic well-being in ECHV	Measuring family economic well-being in ECHV programs
1. Sections of the semi-structured program staff interview protocol (directors and supervisors) and group interview protocol (home visitors)	Perceptions of different family and community facilitators of and obstacles to family economic well-being	Levels of family economic well-being; strategies for supporting the family economic well-being section; professional support and resources needed to address family economic well-being needs; recommendations	Approaches or tools used to measure family economic well-being; recommendations
2. Sections of the semi-structured participant interview protocol and focus group guide	Perceptions of supports and challenges to family economic well-being obtained from community and family characteristics; perceptions of current family economic well-being	Program interactions; perceived changes in family economic well-being; family preferences for strategies that support family economic well-being; recommendations	Perceptions of their own family economic well-being to inform measures

B4. Collection of Data and Quality Control

The study team will designate a lead site liaison for each local ECHV program site. This liaison will conduct all recruitment activities and work with the rest of the site team to conduct all interviews and focus groups at a particular site.

1. Training

To collect high-quality data, we will train teams to conduct the screening, recruiting, interviewing, and focus group activities:

1. Program recruitment and eligibility screening to prepare teams to conduct a broad set of recruitment and eligibility calls to programs (Instrument 1)

2. Program director and supervisor semi-structured interviews to prepare team members to implement the program director interview protocol (Instrument 2) and the supervisor interview protocol (Instrument 3), including selection of questions appropriate for each interviewee

3. Parent participant semi-structured interviews to prepare team members to implement the parent participant interview protocol (Instrument 4), including selection of topics and questions appropriate for each interviewee

4. Home visitor semi-structured group interviews to prepare team members to implement the home visitor group interview protocol (Instrument 5)

5. Parent participant focus groups to prepare team members to implement the parent participant focus group guide (Instrument 6)

Before beginning the recruitment process, we will train site team members on how to establish rapport with program staff, including refusal conversion techniques. Data collection training will include a thorough review of the interview protocols and focus group guides; it will focus on strategies to ensure we collect high-quality data while minimizing burden on respondents, including (1) how to prepare for the interview or focus group (for example, asking only those questions relevant to the participant’s circumstances); (2) how to move efficiently through the interview protocols and focus group guides while collecting high-quality information (for example, how to make decisions about which probes are critical based on answers received to that point in the interview/focus group); (3) common interaction styles for different racial/ethnic groups (for example, use of pausing, nonverbal cues, and eye contact when using video); and (4) how to synthesize notes after each interview/focus group to confirm completeness of the data.

A senior team member will observe at least one interview or focus group conducted by each site team member. As needed, a senior team member will be available to provide additional training to data collectors, such as reviewing training materials or providing them feedback on their interviewing techniques so they can make corrections and apply a more consistent approach to data collection.

2. Recruitment

Section B3 describes the development of the recruitment protocols. To facilitate buy-in, we have also prepared an engaging recruitment packet that includes a letter of support from the Health Resources and Services Administration (HRSA) (Appendix A), a letter of support from the Tribal MIECHV Office (Appendix B) a recruitment email describing the study to the program directors (Appendix C), a frequently asked questions sheet for program staff that includes study details (Appendix D), flyers to encourage parent participants to take part in the interviews and focus groups (Appendices J & K), and a flyer to encourage frontline staff participation (Appendix L).

First, lead site liaisons will email program directors of the 10 local ECHV programs in the recruitment pool, using the recruitment email describing the study (Appendix C), and attach the letters of support from HRSA and the Tribal MIECHV Office (Appendices A & B) and the FAQ sheet (Appendix D). They will then follow up by telephone and use the program recruitment script and eligibility screener to clearly communicate the purpose of the study and the relevant study activities, gauge program’s interest and capacity to participate in the study’s current activities and potentially in a future study. The lead site liaisons will also gather program information so the study team can rate them against the study’s eligibility criteria. Using all of this information, the study team will develop a list of six local ECHV programs to prioritize for data collection, with the remaining programs designated as backups. The study team will then work with the program directors, or their designated representatives, at the six local ECHV programs who agree to participate to identify the supervisors and home visiting staff who will participate in the semi-structured interviews. We will work with the program staff to identify and recruit parents to participate in the focus group or interview.

3. Collecting data

The study team will conduct all data collection activities remotely via telephone and video conferencing software. The study team will also conduct a document review of any materials related to a program’s family economic well-being practices. We will collect these documents primarily via email. For all parent participant interviews, only one study team member will be present to conduct the interview and take notes. With the permission of the parent participants, we will audio and video-record the interviews for later transcription. For all staff interviews and parent participant focus groups, one member of the team will conduct the interview and the other member will take notes. With respondents’ permission, we will also audio and video-record the staff interviews and parent participant focus groups to make sure our notes are complete. The study team will confer after each interview and focus group (using the recordings as needed) to ensure completeness of the data. Throughout the recruitment and data collection period, the whole site team will conduct regular meetings to report and exchange information and strategies, help troubleshoot challenges, and ensure that all data are collected uniformly.

• At each local ECHV program, we will conduct 60-minute semi-structured interviews with up to two program directors (Instrument 2) and 60-minute semi-structured interviews with up to two supervisors (Instrument 3).

• At each local ECHV program, we will conduct 90-minute group interviews with up to six home visitors, interviewing 2-3 home visitors at a time (Instrument 5).

• We will conduct 60-minute semi-structured interviews with parent participants (Instrument 4), speaking to up to fourteen parent participants across up to six local ECHV programs.

• At each local ECHV program, we will conduct a 60-minute focus group with four to six participants who are enrolled in the ECHV program (Instrument 6).

To reduce the burden on staff of participating in the interviews and focus groups, we will be flexible in our scheduling. We will conduct the staff interviews via telephone or video conference, increasing our ability to accommodate staff schedules. We can also break the staff interviews into shorter time frames across multiple days if necessitated by program staff availability. We will hold the interviews and focus groups with local ECHV program parent participants by telephone or video conference at times convenient to them (such as times of day that accommodate different work schedules and availability of child care) to maximize participation.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews and focus groups are not designed to produce statistically generalizable findings, and participation will be wholly at the respondent’s discretion. Response rates will not be calculated or reported.

Nonresponse

As participants will not be randomly sampled and findings are not intended to be representative, we will not calculate nonresponse bias. We will document and report respondent demographics in written materials associated with the data collection.

B6. Production of Estimates and Projections

We will not use the data to generate population estimates, either for internal use or dissemination. This study is intended to present an in-depth description of how ECHV programs and participants define family economic well-being, strategies such programs use to support family economic well-being, and the ways it is measured; the study is not intended to promote statistical generalization to other programs. We do not plan to make policy decisions based on data that are not representative and will not publish population estimates. Information reported will clearly state that results are not meant to be generalizable.

B7. Data Handling and Analysis

Data Handling

We will use the detailed notes and transcriptions for data analysis. Reliance on detailed notes rather than summaries will allow us to include respondents’ own voices and words in the final report (however, we will not attribute any responses to individuals by name in the notes, transcriptions, or report). To ensure the completeness and accuracy of the notes, the data collectors will review the video recordings to fill in any words, phrases, or portions of text that are unclear in the notes. When possible, the team member who conducted the interview or focus group will be the one who listens to the recording and fills in missing text. After using the recording to verify the notes, we will destroy the recording to protect privacy.

Data Analysis

Qualitative data from both the interviews and focus groups will provide comprehensive and rich information for analyzing the ECHV information, strategies, and measures that can best support family economic well-being, and how ECHV staff and participants define such well-being and its components. We will build a structure for organizing and coding the data across all data collection sources to facilitate efficient analysis across respondents. We will supplement the interview and focus group data with relevant information from the programs’ family economic well-being materials collected during the document review.

Coding and analysis approach. Using the same standard procedures for organization, coding, triangulation, and theme identification, we will code and analyze interview and focus group data from each site, as well as any relevant information from the document review. First, we will conduct primary coding, a deductive process in which we apply predetermined codes to the data (Crabtree and Miller 1999). Primary coding allows large amounts of data to be categorized and organized into more manageable portions of text. Next, we will conduct secondary coding (synonymous with the analysis step), applying constant comparative analysis (CCA; Lewis-Beck et al. 2004), an inductive method. We will compare codes or themes that arise from some participants with data from other participants and other identified codes. CCA will enable us to identify themes and categories that emerge from the data rather than searching the data for confirmation of themes or hypotheses. Such themes sometimes will be collapsed into broader themes or will be further refined into subthemes for reporting.

Preparing data sources for coding. First, to prepare for analysis, we will load all interview and focus group notes into NVivo 12 and Dedoose, or a similar qualitative analysis software package, to conduct primary coding. We will create codebooks for a set of deductive (predetermined) codes to be applied to the data sources during primary coding.

We will train a small team of coders to conduct the coding. To facilitate consistency, a lead coder will develop the coding scheme, train the coders, oversee the work, and assess reliability. All coders will code the first transcript together and discuss any differences to establish coder agreement. When new themes or concepts emerge, we will define new codes and apply them to all coded data sources. A senior coder will provide quality assurance by reviewing and providing feedback on at least one coded transcript per coder.

Analyzing coded data. In the next step, we will conduct secondary, or analytic, coding to identify emergent themes and patterns in the data. Using the data coded in NVivo and Dedoose, we will run reports from the software to obtain aggregated output on each primary code across data sources and, using the CCA method, conduct secondary coding of the data. We will refine and summarize emergent themes and findings both within and across data sources. In addition, the lead coder will hold meetings with the coders and other study team members as needed to build consensus around whether specific findings have been classified under the correct thematic labels and confirm that the labels accurately reflect the underlying concept.

Data Use

Summarizing data on specific research questions and concepts to generate findings. When the data have been coded, the team will be able to retrieve and sort data linked to specific research questions and constructs that will improve understanding about the definition of family economic well-being, strategies to support it, and its measurement to better support families. We will synthesize data pertaining to a specific research question across respondents or specific types of respondents. We will describe the strength of the confirming evidence and whether we found disconfirming evidence from any group or participants. Dissemination of findings will include a memo for OPRE and federal partners summarizing findings from the program staff and participant interviews and focus groups. Any materials disseminated will note limitations to the data.

The data will also help inform the project’s conceptual model, identify promising practices to develop and test, and inform other future research efforts. The data may also be used to inform additional activities if they are undertaken, such as using rapid-cycle formative evaluation to assess the fit and feasibility of practices to improve family economic well-being; identifying, adapting, or developing multifaceted measures of family economic well-being that can be used by ECHV programs; and then testing the reliability and validity of such measures.

B8. Contact Persons

The following individuals at ACF and Mathematica are leading the study team:

Pooja Gupta Curtin

Social Science Research Analyst

Office of Planning, Research, and Evaluation

[email protected]

Sarah Avellar

Mathematica

[email protected]

Katie Eddins

Mathematica

[email protected]

Attachments (Instruments)

Instrument 1: Program Recruitment Script and Eligibility Screener

Instrument 2: Program Director Interview Protocol

Instrument 3: Program Supervisor Interview Protocol

Instrument 4: Parent Participant Interview Protocol

Instrument 5: Home Visitor Group Interview Protocol

Instrument 6: Parent Participant Focus Group Guide

Appendices

Appendix A: Health Resources and Services Administration Letter of Support

Appendix B: Tribal Maternal, Infant, and Early Childhood Home Visiting Letter of Support

Appendix C: Program Director Recruitment Email

Appendix D: Study FAQs for Program Staff

Appendix E: Consent Form for Parent Participant Interview

Appendix F: Consent Form for Parent Participant Focus Group

Appendix G: Consent Form for Home Visitor Interview

Appendix H: Consent Form for Program Director Interview

Appendix I: Consent Form for Supervisor Interview

Appendix J: Study Recruitment Flyer for Parent Participant Interview

Appendix K: Study Recruitment Flyer for Parent Participant Focus Group

Appendix L: Study Recruitment Flyer for Frontline Staff

References

Crabtree, B., and W. Miller. “A Template Approach to Text Analysis: Developing and Using Codebooks.” In Doing Qualitative Research, edited by B. Crabtree and W. Miller. Newbury Park, CA: Sage, 1999.

Lewis-Beck, M.S., A. Bryman, and T. Futing Liao. The SAGE Encyclopedia of Social Science Research Methods. Thousand Oaks, CA: Sage Publications, Inc., 2004.

1 Any resulting activities that include information collection that is subject to the Paperwork Reduction Act will be submitted to the Office of Management and Budget (OMB) for review and approval.

2