MIHOPE incentive experiment results: 15-month follow-up

To: Josh Brammer and Margo Schwab, Office of Information and Regulatory Affairs (OIRA), Office of Management and Budget (OMB)

From: Nancy Geyelin Margie and Laura Nerenberg, Office of Planning, Research and Evaluation (OPRE), Administration for Children and Families (ACF)

Date: February 2019

Re: MIHOPE incentive experiment results: 15-month follow-up (OMB Control Number: 0970-0402)

This memo reports on the results of the incentive experiment conducted during the MIHOPE data collection round when participating children were about 15 months old. OMB approved this incentive experiment on April 22, 2015. The memo contains the following sections:

• An introduction to the 15-month data collection activities and purpose of the experiment

• The incentive experiment design and sample

• Effects on overall response rates at the end of the data collection period

• Effects on differential response rates at the end of the data collection period

• Effects on overall response rates at the end of the early bird period

MIHOPE 15-month Follow-up Data Collection

Initial Plans for Data Collection

The Mother and Infant Home Visiting Program Evaluation (MIHOPE) is a longitudinal study providing information about the effectiveness of the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program. Families enrolled in the study between 2012 and 2015. When children in the MIHOPE sample were approximately 15 months old, families were contacted and invited to participate in a 60-minute telephone interview and a 90-minute in-home visit. Initial communications to participants included information about both the phone interview incentive and in-home visit incentive. Respondents were first invited to complete the phone interview. If families had not completed the telephone interview within four weeks, field staff followed up with them to invite them to complete the phone interview and the in-home visit (this is referred to as in-person locating). The data collection period for each family was approximately 22 weeks long.

Families were initially offered gift cards valued at $25 for completing the interview and $20 for completing the in-home visit activities.

Addition of Incentive Experiment

Response rates for cases released during the first four months of data collection were below the target of 85% for both the phone interview and the in-home data collection. For example, 66% of 191 cases fielded in April through July 2014 had responded to the follow-up phone interview and 63% had provided some data through the in-home assessments.

In addition, the percentage of mothers requiring in-person locating was much higher than expected during the first portion of the fielding period. Initially, the study planned to try to reach mothers by phone for four weeks and then, if they did not respond, use field staff to try to locate them for approximately four months. The project assumed that about half of mothers would complete the phone interview by responding to telephone calls or by calling the Mathematica Survey Operations Center and the remainder would require in-person location by field staff. However, during the first portion of the fielding period, only approximately 30% of mothers completed the phone interview via phone, requiring field staff to locate and facilitate phone interview completion for an additional 55% in order to reach the 85% target response rate. This resulted in increased data collection costs.

We hypothesized that increasing the incentives would increase early response rates and decrease the need for in-person locating. Therefore, we proposed systematically studying whether increased incentives and/or an early bird incentive would impact response rates and need for in-person locating.

Incentive Experiment Design

OMB approved conducting this experiment in April 2015. The experiment sample included the 1,187 families participating in MIHOPE who had children approximately 15 months old between July 2015 and April 2016. The experiment used a factorial design,¹ and families were randomly assigned to one of six experimental groups.

For the phone interview, families were offered:

1. the standard incentive amount ($25),

2. a higher incentive amount ($35), or

3. an early bird incentive ($35 offered in the first four weeks of data collection and $25 offered after that).

For the in-home visit, families were offered:

1. the standard incentive amount ($20) or

2. the higher incentive amount ($40).

The combination of these offered incentives yielded the following six groups presented here as Table 1:

Incentive Experiment Condition	Phone Interview	In-home Visit	Number of Families Randomly Assigned	Percentage of Experiment Sample
1A	Standard	Standard	198	16.68
2A	Higher	Standard	199	16.76
3A	Early bird	Standard	199	16.76
1B	Standard	Higher	201	16.93
2B	Higher	Higher	194	16.34
3B	Early bird	Higher	196	16.51

The total amount offered to each group is as follows here in Table 2:

		In-home visit
		Standard $20	Higher $40
Phone interview	Standard $25	Group 1A ($45)	Group 1B ($65)
	Higher $35	Group 2A ($55)	Group 2B ($75)
	Early bird $35 or $25	Group 3A ($55 or $45)	Group 3B ($75 or $65)

Effects on Overall Response Rates at the End of the Data Collection Period

For the experiment sample, the overall response rate for the interview was 84%. The data collection period was 22 weeks long. The design of the experiment allows us to examine both main effects and interaction effects, but the main effects were the primary focus in this experiment. The term “main effects” refers to the effect of the phone interview incentive, ignoring the effects of the in-home interview incentive (for example, the effect of the higher phone interview incentive amount compared to the standard phone interview incentive amount, regardless of the in-home incentive amount). The term “interaction effects” refers to the effect of the phone interview incentive dependent on the level of the in-home visit incentive (for example, the effect of the higher phone interview incentive amount when the higher in-home incentive amount was offered).

Main Effects

Table 3

Response Rates at the End of the Data Collection Period: Main Effects

Incentive type	Group offered incentive	Group not offered incentive	Significance
Phone Interview Response Rates
Higher phone interview amount	86%	81%	*
Early bird phone interview incentive structure	83%	81%
Higher in-home visit amount	85%	83%	*
In-Home Visit Response Rates
Higher phone interview amount	82%	77%
Early bird phone interview structure	77%	77%
Higher in-home visit amount	79%	78%

Significance levels: *** p < .01, ** p < .05, * p < .10

Phone Interview Response Rates

The effects of the various incentives on phone interview response rates are shown in Table 3. The higher phone interview incentive amount increased the phone interview response rate by about 5 percentage points. However, the early bird phone interview incentive did not have a statistically significant impact on response rates. Finally, the higher in-home incentive did have a statistically significant impact on phone interview response rates.

In-home Response Rates

As shown in Table 3, neither the in-home incentive amount nor the phone interview incentive amounts had much effect on in-home response rates.

Interaction Effects

Table 4

Interaction Effects between the Phone Interview Incentive and the In-Home Incentive

on Phone Interview Response Rates

Incentive type	Impact when higher in-home incentive offered	Impact when standard in-home incentive offered	Significance
Higher phone interview amount	+5 percentage points	+5 percentage points
Early bird phone interview structure	+4 percentage points	+1 percentage points

Significance levels: *** p < .01, ** p < .05, * p < .10

Unlike the main effects, no interactions between the phone interview incentive and in-home incentive were statistically significant. These effects are shown in Table 4. Notably, the higher phone interview incentive appears to have been effective regardless of the in-home incentive offered, producing response rates approximately five percentage points higher than the standard interview incentive across in-home incentive conditions.

Effects on Differential Response Rates at the End of the Data Collection Period

Effects on Response Rates for Subgroups

The MIHOPE incentive experiment was designed to increase the study’s overall response rate, maintaining the statistical power of the baseline study. Incentives were not targeted to particular demographic groups in this experiment. However, as part of OPRE and OMB’s effort to understand how incentives affect the representativeness of phone interview responses, we analyzed the experimental data by key demographics of interest in the MIHOPE population. Neither the higher phone interview incentive nor the early bird incentive significantly increased response rates for demographic subgroups (as measured at study entry) that were less likely to respond to the 15-month data collection efforts.

Though only two of the effects by subgroup reached the threshold of statistical significance, Table 5 indicates that the higher phone interview incentive and early bird incentive consistently induced greater changes in response rates for the groups that were more likely to respond to data collection efforts.

Table 5

Phone Interview Response Rates for Family Baseline Characteristics Subgroups

	Percent of Baseline Study Populationa	Response Rates				Significance
		Total	Standard Incentive	Higher Incentive	Early Bird Incentive
Not pregnant at study entry	24%	89%	83%	93%	91%	**
Pregnant at study entry	76%	82%	81%	84%	81%
Did not move in the prior year	47%	85%	82%	90%	84%	*
Moved in the prior year	53%	83%	81%	83%	84%
Father figure in household	45%	87%	83%	90%	87%
Father figure not in household	55%	81%	80%	83%	81%
Married to biological father	18%	92%	89%	93%	94%
Not married to biological father	82%	82%	80%	85%	82%

^aReflects the proportion of these groups within the incentive experiment sample.

Effects on Response Rates for the Program and Control Groups

Because MIHOPE is a randomized controlled trial, we were interested in whether the alternative incentives had bigger effects for the study’s program group or control group. Although the higher phone interview incentive and the early bird phone interview incentive increased phone interview response rates slightly more for the program group (shown in Table 6), the difference in effects between the program and control groups are not statistically significant. In addition, response rates for the program group and control group within each incentive experiment group are not statistically significantly different from each other.

Table 6

Phone Interview Response Rates for Program and Control Groups

	Standard Incentive	Impact of Higher Incentive	Impact of Early Bird Incentive	Significance
Program Group	80%	+7 percentage points	+4 percentage points
Control Group	83%	+3 percentage point	+0 percentage points

Effects on Overall Response Rates at the End of the Early Bird Period

Respondents received the early bird phone interview incentive only if they completed the phone interview in the first four weeks. If the early bird incentive was effective, it would reduce the cost of conducting the phone interviews by reducing the effort the study team had to put into getting families to respond. We examined response rates at the end of the first four weeks of data collection to determine whether the early bird incentive increased early responses.

As shown in Table 6, at the end of the four-week early bird period, respondents in the early bird incentive group had a slightly higher phone interview response rate than those who were offered the standard phone interview incentive, but this difference was not statistically significant. The early bird phone interview incentive did not effectively encourage early responses.

Our design of the incentive experiment allowed us to examine whether the null effect might be attributed to the early bird incentive structure or the early bird incentive amount. The offer of $35 was time limited for the early bird phone interview group, but was available for the full data collection period for the higher phone interview incentive amount group. Table 7 shows that the offer of the time-unlimited higher phone interview incentive appears to have been more effective than the early bird incentive (the offer of $35 that was time-limited) – the response rate for the group offered the higher amount is about 7 percentage points higher than for the group offered the standard incentive and the effect is statistically significant. Because the early bird incentive was offered only for the phone interview, effects on in-home response rates are not presented in Table 7.

Table 7

Phone Interview Response Rates at the End of the Early Bird Period

Incentive type	Group offered incentive	Group not offered incentive	Significance
Early bird phone interview incentive structure	49%	45%
Higher phone interview amount	52%	45%	**

Significance levels: *** p < .01, ** p < .05, * p < .10

Conclusion

This memo provides information on the results of an incentive experiment conducted at the MIHOPE follow-up that occurred around the time children were 15 months old. The experiment used a factorial design to test three incentive conditions for the phone interview and two incentive conditions for the in-home visit.

At the conclusion of the experiment, we found that participation in the phone interview was responsive to variation in incentives. The interview response rate was increased by both a higher phone interview incentive and a higher in-home visit incentive. Contrary to expectation, the early bird incentive structure did not effectively increase phone interview responses during initial weeks of data collection. The in-home study response rate was not meaningfully affected by either the in-home incentive amount nor the phone interview incentive amount.

As a randomized controlled trial of an intervention model, MIHOPE was designed to detect meaningful effects of program participation on family and child outcomes over time. The goal of the MIHOPE incentive experiment was to increase the study’s overall response rate, preserving the statistical power of the 15-month data collection. However, as part of OPRE’s efforts to examine whether incentives affect the representativeness of survey responses, we also examined the effects of the experiment on response rates by key demographic subgroups. The incentive conditions tested did not appear to increase phone interview responses from groups who were less likely to respond to data collection efforts. Incentives did not produce statistically significant differences in response rates by treatment/control grouping, or for subgroups of families defined by their baseline characteristics.

1 A factorial design allows researchers to examine how two or more factors (independent variables), each with discrete levels, affect a dependent variable, both independently and together.

6