MIHOPE-3G_Supporting Statement B

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Mother and Infant Home Visiting Program Evaluation: Third Grade Follow-Up (MIHOPE-3G)

OMB Information Collection Request

0970 - 0402

Supporting Statement

Part B

July 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Nancy Geyelin Margie

Laura Nerenberg

Part B

For information about statistical methods and related details for previous phases of MIHOPE, see previously-approved information collection materials:

• MIHOPE 1: approved July 12, 2012

• MIHOPE 2: approved June 26, 2013

• MIHOPE Check-in: approved August 6, 2015

B1. Objectives

Study Objectives

In 2011, the Administration for Children and Families (ACF) and the Health Resources and Services Administration (HRSA) within the U.S. Department of Health and Human Services (HHS) launched the Mother and Infant Home Visiting Program Evaluation (MIHOPE). MIHOPE is providing information about the effectiveness of the Maternal, Infant, and Early Childhood Home Visiting Program (MIECHV) in its first few years of operation and providing information to help states and others develop and strengthen home visiting programs in the future. The goals of the study are:

1. to understand the effects of home visiting programs on parent and child outcomes, both overall and for key subgroups of families,

2. to understand how home visiting programs were implemented and how implementation varied across programs, and

3. to understand which features of local home visiting programs are associated with larger or smaller program impacts.

Generalizability of Results

This randomized study is intended to produce internally-valid estimates of the causal impact of home visiting, not to promote statistical generalization to other sites or service populations. This study could help ACF, HRSA, and the broader home visiting field understand the long-term impact of home visiting on low-income families.

Appropriateness of Study Design and Methods for Planned Uses

The purpose of this data collection activity is to help us understand the long-term effects of home visiting. Evaluating the effect of home visiting on families’ outcomes and the pathways through which home visiting affects families’ long-term outcomes would not be possible without following up with families at multiple time points (see Supporting Statement A.2 Other Data Sources and Uses of Information).

A follow-up approximately at third grade allows the study team to obtain school records that include reading and math test scores at the first point at which they become available. Third grade reading scores in particular have been associated with high school graduation rates. Furthermore, obtaining information at or following third grade is important for conducting benefit-cost analyses. Because benefit-cost analyses weigh the cost of operating a program with subsequent monetary benefits to the government, to society, and to families that might result from participation in the program, allowing more time for benefits to accrue will provide more precise estimates of these benefits. The study team identified seven areas of adult and child functioning and behavior where effects of home visiting services are likely to be observed in administrative data when children are of third grade age:

• Maternal health

• Child health

• Child development and school performance

• Child maltreatment

• Parenting

• Crime or domestic violence

• Family economic self-sufficiency

A study that follows families over time provides an opportunity to examine child and family outcomes at individual time points as children get older, and to learn about the trajectories of child and family outcomes.

Given the study’s requirements for local programs, the home visiting programs participating in MIHOPE are not representative of all MIECHV local programs. It is not clear how the effects of home visiting where it was studied in MIHOPE would compare with the results for MIECHV as a whole. Key limitations will be included in written products associated with the study. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

At baseline, MIHOPE recruited 4,229 families from 88 local programs (sites) in 12 states. Families were randomly divided between a program group, which could be enrolled in one of the home visiting programs being studied, or a control group, which was provided with referrals to other services in the community. Families were eligible for the study if (1) the mother was pregnant or the family had a child under six months old when they were recruited for the study, (2) the mother was 15 years or older at time of entry in the study, and (3) the mother was available to complete the baseline family survey.

The study plans to conduct third grade follow-up activities with most families who enrolled in the study, not just those who have completed previous rounds of follow-up data collection. The sample eligible for third grade follow-up data collection is different from the baseline MIHOPE sample because 1) some families withdrew from the study and 2) some mothers had miscarriages after entering the study while pregnant or the MIHOPE child was no longer alive by the time they were contacted for the 15-month follow-up.

As shown in Figure B.1, the study enrolled 4,229 families, all of whom completed the baseline interview. Between the baseline interview and the 15-month follow up, 11 families withdrew from the study, resulting in a fielded sample of 4,218 families at the 15-month follow up. At the 15-month follow-up, we learned that 103 children had never been born (for example, because of miscarriages) or had died soon after birth (by the 15-month follow-up point) and, therefore, these families were not contacted for subsequent data collection efforts. Between the 15-month follow up and the 2.5-year and 3.5-year Check-in and subsequent kindergarten follow-up, two families also withdrew from the study. Thus, the 2.5-year and 3.5-year Check-in and kindergarten follow-ups included a fielded sample of 4,113 families. Between the kindergarten follow-up and third grade follow-up, 8 families withdrew from the study. Therefore, the sample eligible for the third grade follow up includes 4,105 families.

Figure B.1 – MIHOPE Sample at Baseline, 15-Months, 2.5-Years, 3.5-Years, Kindergarten, and Third-Grade Follow-Ups

Baseline Sample:

4,229 families

Sample Eligible for Third-Grade Follow-Up:

4,105 families

Minus 8 families who withdrew from the study

Sample Fielded at 2.5- and 3.5-Year Check-in and Kindergarten Follow-Up:

4,113 families

Minus 103 mothers with miscarriages or with a child who died shortly after birth and two families who withdrew from the study

Sample Fielded at 15-month Follow-Up:

4,218 families

Minus 11 families who withdrew from the study

Sampling and Site Selection

Local sites meeting several criteria were chosen to participate in the study: (1) operating programs that existed for at least two years by the time of study recruitment, (2) evidence of enough demand for home visiting services that they could provide a control group, (3) no evidence of severe implementation problems that would interfere with the program’s ability to participate in the study, and (4) a contribution to the diversity of sites and families for purposes of estimating effects for important subgroups of families.

To estimate the effects of home visiting on family outcomes, MIHOPE enrolled over 4,200 families across 88 sites in 12 states. Families were eligible for the study if they included a pregnant woman or an infant under six months old and the mother was at least 15 years old at the time of study entry. Families were recruited into the study by Mathematica’s survey research staff, who visited families to obtain informed consent when home visitors determined whether a family was eligible for the study or soon after that determination had been made. For state child welfare records and school records, we are following up with 11 of the 12 original states where MIHOPE enrolled participants (the study obtained child welfare records from all 12 MIHOPE states at the 15-month follow-up).¹

B3. Design of Data Collection Instruments

As noted above, we plan to continue to obtain child welfare data from 11 state agencies and plan to request school records data from state and local education agencies. (For school records, we have assumed that we will obtain data from up to 11 states and 37 local education agencies.²)

For this round of administrative data collection, we are expecting either two or three data extracts from each state or local education agency (as noted in the burden table of Supporting Statement A, Table 2) because the children in the study reach their third grade year during four different academic years (from 2021-22 through 2024-25). To be able to abide by the study’s reporting timeline, one “interim” data extract will be requested from most agencies providing administrative data, although a second “interim” file will be needed for one agency due to data expungement. Additionally, a “final” data extract will be requested from all data providers. The “interim” file(s) will allow the study team to conduct preliminary data processing and analyses with earlier cohorts of children, while the “final” data file will include the fourth cohort of children and will provide complete data for the third grade analysis. We have assumed a lower burden per state for child welfare records (as compared to school records). Requests to agencies for MIHOPE-3G efforts are included in Instruments 1-4.

B4. Collection of Data and Quality Control

MIHOPE’s third grade follow-up will be based on administrative data alone. MDRC will be collecting the data from state child welfare agencies and state and local education agencies. These data will be transferred electronically via secure and efficient means, such as through use of the file transfer platform, Box. After reviewing data files received from these agencies, MDRC will perform quality checks to see if the data are as expected and will reach out to the data provider if questions arise to work to resolve them.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Administrative data will be collected from states and local school districts in 11 of the 12 states where the local programs in MIHOPE operated during study enrollment. Since children in the MIHOPE sample will be in approximately third grade when administrative data collection for this follow-up will take place, some families may no longer reside in these 11 states and therefore may not be included in the administrative data the study team collects. In addition, the study team may not obtain data for some study families because of missing or incorrect SSNs (which are used to obtain administrative data). Efforts will be made to attempt to collect as much as administrative data on the study sample from states and local education agencies as feasible within the scope of the study.  

NonResponse

Following the collection of administrative data, a non-response analysis will be conducted to determine whether the results of the study may be biased by not obtaining administrative data for all study participants.

In particular, two types of bias will be assessed: (1) whether estimated effects among participants for whom administrative data has been acquired apply to the full study sample, and (2) whether participants from the program group for whom administrative data has been acquired are similar to control group members for whom administrative data has been acquired. The former type of bias affects whether results from the study can be generalized to the wider group of families involved in the study, while the second assesses whether the impacts of the programs are being confounded with pre-existing differences between the sample for whom administrative data have been collected for the program group and control group. 

 

To assess non-response bias, several tests will be conducted: 

• The proportion of program group and control group members will be compared to make sure the match rate to administrative data was not significantly higher for one research group, if the data source is expected to have similar match rates.³

 

• A logistic regression will be conducted among sample members with administrative data. The “left hand side” variable will be their assignment (program group or control group) while the explanatory variables will include a range of baseline characteristics. An omnibus test such as a log-likelihood test will be used to test the hypothesis that the set of baseline characteristics are not significantly related to whether a sample member with administrative data is in the program group. Not rejecting this null hypothesis will provide evidence that program group and control group members with administrative data are similar.  

 

• Baseline characteristics of sample members with administrative data will be compared to baseline characteristics of those who did not match to administrative data. This will be done using a logistic regression where the outcome variable is whether someone matched to an administrative data source, and the explanatory variables are baseline characteristics. An omnibus test such as a log-likelihood test will be used to test the hypothesis that the set of baseline characteristics are not significantly related to whether someone matched to an administrative data source. Not rejecting this null hypothesis will provide evidence that sample members who matched versus did not match to an administrative data source are similar.  

 

If any of these tests indicate that non-match is providing biased impact estimates, a standard technique such as multiple imputation or weighting by the inverse probability of match will be used to determine the sensitivity of impact estimates to non-match. 

   

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination. 

 

The impact estimates produced are for official external release. Given the study’s requirements for local programs, the home visiting programs participating in MIHOPE are not representative of all MIECHV local programs. It is not clear how the effects of home visiting where it was studied in MIHOPE would compare with the results for MIECHV as a whole. These limitations of the study will be included in resulting materials, as appropriate. 

  

Methods to be used for statistical tests and analytical techniques that will be used: 

  

Impact Analysis  

The impact analysis will assess the effectiveness of early childhood home visiting programs in improving the outcomes of families and children when children are in third grade, both overall and across key subgroups of families and programs. Random assignment was used in MIHOPE to create program and control groups that were expected to be similar in all respects when they entered the study. As is standard in random assignment studies, the primary analytical strategy in MIHOPE-3G will be to compare the outcomes of the entire program group with those of the entire control group (an “intent-to-treat” analysis). Doing so preserves the integrity of the random assignment design and means that any differences that emerge after random assignment can be reliably attributed to the program group’s access to evidence-based home visiting. 

 

Information on sample members’ baseline characteristics will be used in the analysis to increase the precision of estimated impacts.  

 

To address the question of whether home visiting programs have larger effects for some groups of families, effects can also be compared across key subgroups of families. This approach is consistent with the MIHOPE analysis that was conducted when children were 15 months of age and follows the recommendations of Bloom and Michalopoulos (2013). For example, in estimating the effects for mothers who were pregnant and the effects for those whose children were infants when they entered the study, the impact analysis would investigate whether estimated effects were larger for one group than for the other. If there are no statistically significant differences in the estimated impacts across subgroups and there are statistically significant effects estimated for all families, the presumption would be that home visiting is effective for all subgroups. This approach is proposed because estimated effects for subgroups are less precise than estimated effects for the full sample (because subgroup sample sizes are smaller than the full sample), meaning that it is likely that estimated effects for some subgroups would not be statistically significant even if the program were modestly effective for that subgroup. 

Benefit-Cost Analysis  

Benefit-cost analyses represent a way of assessing whether social programs are an efficient use of governmental resources. Given the budgetary constraints of publicly funded programs and the diverse needs for them, a benefit-cost analysis can help to identify programs or policies that provide the highest returns on public investments with the goal of improving the lives of program participants and other members of society. The MIHOPE-3G benefit-cost analysis will use information collected through the time the focal children are in approximately third grade to weigh the cost of operating a program with subsequent monetary benefits to the government, to society, and to families that might result from participation in the program. Information on program costs were collected and analyzed during earlier stages of MIHOPE; the cost data were derived from (i) total program expenditures for one calendar year, classified into cost categories such as personnel and supplies; (ii) service delivery data for the same time period as local program expenditures; and (iii) MIHOPE family service logs.

The team plans to post analysis plans at clinicaltrials.gov, as they have for earlier MIHOPE data collection efforts.

B7. Data Handling and Analysis

Data Handling

Administrative data will be checked after receipt to review for possible data errors or aspects of data file receipt that the study team has questions about. Data processing will be conducted by an experienced group of staff from the study team. Quality checks will be conducted to ensure that administrative data processing was conducted as expected.

Data Analysis

Since this is an impact study and will include a benefit-cost analysis, this information is provided under Production of Estimates and Projections (B6).

Data Use

A report published by the federal government will show estimated long-term effects of home visiting programs and compare their benefits and costs through the third grade follow-up time point. This report will be written to inform ACF and HRSA, as well as to provide information to the broader home visiting field. Other interested parties, such as home visiting model developers, states, territories, Tribes, and Tribal organizations that oversee the implementation of home visiting programs, local home visiting programs, and researchers in home visiting will all have potential interest in incorporating the findings into their work. Limitations to the data will be included in the published report.

 

We plan to archive the data collected for the third grade follow-up at the Inter-university Consortium for Political and Social Research (ICPSR), if allowed by the administrative data agency that provided the data. Data from earlier MIHOPE data collection efforts is also being archived at ICSPR, and the team has prepared extensive documentation to guide other researchers in using that data. Additional documentation describing the MIHOPE elementary school follow-up data will be prepared as part of the data deposit.   

 

B8. Contact Persons

• MDRC

  • Kristen Faucetta ([email protected])

  • Charles Michalopoulos ([email protected])

• Nancy Geyelin Margie (OPRE/ACF)

• Laura Nerenberg (OPRE/ACF)

Attachments

Current Request – MIHOPE-3G Materials

Instrument 1_School Records Request From Existing Data Provider

Instrument 2_School Records Request From New Data Provider

Instrument 3_Child Welfare Records Request From Existing Data Provider

Instrument 4_Child Welfare Records Request From New Data Provider

Appendix A_MIHOPE_IRB Approval Letter

Previously Approved and Ongoing with No Changes – MIHOPE-K Instruments

Instrument 5_MIHOPE-K Child Welfare Records Request

Instrument 6_MIHOPE-K School Records Request

1We are following up with agencies in 11 out of the 12 states from which we recruited the MIHOPE sample because the California state IRB did not approve collection of data from state or local agencies beyond MIHOPE’s 15-month follow-up point.

2We are factoring in the ability to request data from more local education agencies at the third grade follow-up point than we did at the kindergarten follow-up due to the importance of student test scores at third grade (student standardized test scores are not consistently available from states and local education agencies before third grade).

3For example, with school records data, the expectation would be that both research groups would be enrolled in school at a similar rate; with other data sources, for example public benefits, the enrollment rate might not be the same across research groups due to the possibility of home visiting having an effect on public benefit receipt.  

9