NonSub Change Memo - Changes to Incentive for 3.5yr old round

To: Stephanie Tatham, Office of Information and Regulatory Affairs (OIRA); Office of Management and Budget (OMB)

From: Nancy Geyelin Margie, Office of Planning, Research and Evaluation (OPRE); Administration for Children and Families (ACF)

Date: July 25, 2016

Subject: Interim experiment results and non-substantive change request for MIHOPE family follow-up incentive structure (Information Collection 0970-0402)

As approved in our information collection request for the MIHOPE follow-up data collection (OMB Control No: 0970-0402, approved 08/06/2015), we are conducting an experiment with the level and timing of incentives, in order to inform future data collection efforts for MIHOPE and other studies. Interim analyses based on 688 participants (final sample will be approximately 1700 participants) suggest that the early bird incentive is generating a higher response rate to obtain survey responses. Therefore, we request approval to implement the early bird incentive with all participants when we survey the families when the children are 3½ years old.

Experiment Structure

Because MIHOPE participants are highly mobile, it has been a challenge to achieve acceptable response rates for follow-up data collection in MIHOPE. To examine the best method for maximizing survey response, we are testing two commonly used incentive structures: prepaid incentives (through which individuals receive a small payment when they are notified of the survey and a larger payment when the survey is completed) and “early bird” incentives (through which individuals receive a larger payment if they complete the survey within a set period of time after being notified of the survey). Researchers have long regarded prepaid incentives as having the potential to generate increased response rates to surveys (Cantor et al. 2008; Singer et al. 1999). Furthermore, early bird incentives have been shown to decrease the number of days to complete a survey, which can lead to a decrease in the total survey field period and potentially result in lower costs (LeClere et al. 2012).

As shown in Table 1, the experiment divided individuals at random into four groups: (1) a group that receives a prepayment, (2) a group that receives an early bird incentive, (3) a group that receives both a prepayment and an early bird incentive, and (4) a control group that receives neither.

Table 1: Experiment Conditions

Early bird incentive	Prepaid incentive
	No	Yes
No	$15 after completing the survey	$5 with advance letter, $10 after completing the survey
Yes	$25 if survey completed within 8 weeks, $15 otherwise	$5 with advance letter, remainder ($20 if survey completed within 8 weeks, $10 otherwise) after completing the survey

The early bird incentive is available for the first eight weeks of the data collection period because after that point, field locating begins for cases that have not already responded to the survey. We are particularly interested in maximizing response rates before field locating begins, since data collection efforts become much more costly at that point in the data collection period. In addition, the project saves funds when participants respond quickly, because resources aren’t needed to continue to track them down. This provides more funds to track down participants who are most difficult to find, thereby increasing response rates.

Interim Experiment Results

The following table shows the response rates in each of the four experiment groups at the end of week 8 for the combined sample of the first two sample releases of the 2½-year-old survey round (n=688):

Table 2: Response Rates through Week 8

Experimental Group	Response Rate
Control	43%
Prepay	41%
Early Bird	52%
Prepay + Early Bird	48%

Although the difference in response rates across all of the groups is not statistically significant with the current sample, the factorial design of the experiment allows us to isolate the effects of the early bird incentive and the prepay incentive at this time. The effect of the early bird incentive can be estimated by comparing the average outcome for the two groups with the early bird incentive to the average outcome for the two groups that do not have the early bird incentive (50.2-42=8.1). This analysis shows that the early bird incentive increased the response rate by 8.1 percentage points, and this difference is statistically significant (p-value=.03). For the prepay incentive, comparing the average outcome for the two groups with the prepay incentive to the average outcome for the two groups that do not have the prepay incentive (44.6-47.5= -2.9) indicates that the prepay incentive has not generated higher response rates, and the difference (-2.9) is not statistically significant (p-value=.45).

Given these results, we request permission to offer the early bird incentive ($25 if survey completed within 8 weeks, $15 otherwise) to all participants in the 3½-year-old survey round.

Once the incentive experiment has concluded, we will share the final results with OMB. If the final analyses result in the same finding as the interim analyses, we will request permission to implement the early bird incentive with all participants in the 2½-year-old group who have not yet been contacted. However, if the final analyses do not support the continued use of the early bird incentive, we will request to discontinue its use with both the 2½ -year-old and 3½-year-old groups.

REFERENCES

Cantor, D., B. O’Hare, and K. O’Connor. “The Use of Monetary Incentives to Reduce Non-Response in Random Digit Dial Telephone Surveys.” In Advances in Telephone Survey Methodology, edited by J.M. Lepkowski, C. Tucker, J.M. Brick, E. De Leeuw, L. Japec, P.J. Lavrakas, M.W. Link, and R.L. Sangster, pp. 471–498. New York: J.W. Wiley and Sons, Inc., 2007.

LeClere, F., S. Plumme, J. Vanicek, A. Amaya, and K. Carris. “Household Early Bird Incentives: Leveraging Family Influence to Improve Household Response Rates.” American Statistical Association Joint Statistical Meetings, Section on Survey Research, 2012.

Singer, Eleanor, John Van Hoewyk, Nancy Gebler, Trivellore Raghunathan, and Katherine McGonagle. “The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys.” Journal of Official Statistics, vol. 15, 1999, pp. 217–230.



3