Memo for Incentives Experiment 12-21-17v2

United States

Department of

Agriculture

Food and

Nutrition

Service

3101 Park

Center Drive

Alexandria, VA

22302-1500

Date: December 21, 2017

From: Rosemarie Downer, Ph.D., Social Science Researcher; USDA, Food and Nutrition Service (FNS), Office of Policy Support

Through: Rachelle Ragland-Greene Information Collection Officer; FNS, Planning and Regulatory Affairs Office rgreeene

To: Steph Tatham, OMB/OIRA Desk Officer

Subject: ICR Ref. No. 201702-0584-002 - Proposed Survey Incentives Experiment

In response to OMB’s concern about providing a $5 pre-incentive to the entire study sample in the absence of evidence showing that a pre-survey incentive reduces non-response bias in SNAP participants, FNS is proposing to conduct an experiment within this study. The proposed experiment would test the effectiveness of survey pre-incentives in combination with post-incentives, as a way to improve survey representativeness and add to the evidence to inform use of incentives in future studies on program participants.

The proposed sample size for this study is 6,593. In the original proposal, we had planned to send out a $5 pre-incentive with all surveys, and provide a $20 post-incentive for all completed surveys. We now propose to randomly assign the sample to one of two groups to test the combination of pre- and post-survey incentives. The design and method would be as follows:

We would sample 6,593 SNAP recipients, as originally proposed. We would then randomly assign recipients to one of two conditions:

1. No pre-incentive; $20 post-incentive for completed surveys;

2. $2 pre-incentive; $20 post-incentive for completed surveys.

We calculate that each incentive subgroup would have an effective size of 2,746, with a design effect of 1.2. At this size, minimum detectable differences could be seen at a range of subgroup response rates, at both a 0.1 and a 0.05 significance level, as shown in the tables below.

Table 1. Minimum detectable difference for response rates between two subgroups (0.1 level of significance)

Subgroup response rate (R)	30%	40%	50%	60%	70%	80%	90%
Minimum detectable difference	2.66%	2.82%	2.86%	2.79%	2.59%	2.24%	1.65%
Subgroup sample size = 3296. Assuming a one-sided test with a significance level of 0.1, power of 0.80, and DEFF = 1.2.

Table 2. Minimum detectable difference for response rates between two subgroups (0.5 level of significance)

Subgroup response rate (R)	30%	40%	50%	60%	70%	80%	90%
Minimum detectable difference	3.12%	3.31%	3.35%	3.26%	3.03%	2.62%	1.92%
Subgroup sample size = 3296. Assuming a one-sided test with a significance level of 0.05, power of 0.80, and DEFF = 1.2.

We would then analyze the response rates to determine the effectiveness of including a pre-incentive with a post-incentive on response rates. As subgroup response rates allow, we will examine the effect of pre-incentives on response rates within additional strata, including race; gender; education level; and household size.

This experiment will provide valuable evidence for future studies. To date, no study has examined the effectiveness of providing no pre-incentive in combination with a post-incentive versus a small pre-incentive in combination with the post incentive on survey response rates of SNAP populations, or on any population of program recipients. Evidence from broader samples suggests that pre-incentives increase response rates and survey representativeness substantially, but this effect has not yet been tested in SNAP populations.¹ The results of this experiment will thus help in designing representative studies of SNAP populations and other program recipient populations in the future.

1 Mercer, A, Caporaso A, Cantor D, Townsend R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79: 105-129.

2