ERS respose to omb passback

From OMB:

If all participants will be using the same incentive scheme, it’s not clear how ERS will be able to judge the effectiveness of the ‘gamification’.

Please calculate the minimum detectable difference for a randomly split sample that uses the described incentive structure as well as a more traditional incentive structure.

Alternatively, explain how the results from this pilot might be compared to previous collection efforts in order to estimate the effectiveness of the proposed incentive structure.

A split experiment that randomly assigns respondents to either the new incentive scheme or a more traditional incentive structure offers clear findings on the effectiveness of the new incentive scheme. However, the focus of this Pilot study is to explore the feasibility of an alternative data collection methodology that exploits the latest computer technologies and the increasing ownership of smartphones instead of testing the effectiveness of a different incentive scheme. An incentive experiment that would bring an additional layer of complexity to the already complicated design is not planned for this Pilot study.

That being said, the Pilot study can still allow us to learn about the performance of the new incentive scheme on the completion of Food Log, especially the performance of the incentive scheme for different kinds of households. The current incentive scheme consists of two parts. The first part is at the household level; households earn a $50 incentive for completing the Initial Interview as well as a $5 for completing the Income Worksheet. This part doesn’t involve any gaming mechanism and the total dollar amount is the same for all households. The second part of the incentive scheme is at the person level and builds on the gamification theory. Specifically, each household member is given a $3 incentive for each day that they complete the Food Log, which means that each household member can earn up to $21 throughout the data collection week. In addition, a household is given a $50 bonus for completing the Final Interview AND for all household members completing the Food Log for all seven days. The accumulation of incentive amounts is conveyed to household members each time they log in to the Food Log and is also conveyed to the individual household members by text messages and/or emails. The Pilot Study will allow us to examine how household members respond to this part of the incentive scheme.

We will look at the effects of the second part of incentive scheme on Food Log completion from two aspects. First, the 12 PSUs drawn for the Pilot Study are part of the FoodAPS-1 PSUs. We will compare data from the Pilot Study to data for the same 12 PSUs from FoodAPS-1 using three metrics (the proportion of households where at least one person completed the Food Log, the proportion of households where everybody has completed the Food Log, and the proportion of individual household members completing the Food Log). We will also examine these proportions by meaningful household characteristics (such as single-person households, small households, or large households) using the Pilot Study data and the FoodAPS-1 data. This set of analyses yields information on how the current incentive scheme works in relation to the more traditional scheme used for FoodAPS-1.

Second, we will compare the characteristics of households and individual household members who completed the Food Log for all seven days to those who didn’t do all seven days. This set of analyses will allow us to learn who are responsive to the gamification of the incentive scheme and to make recommendations for FoodAPS-2.

From OMB:

The requested incentive amounts are large, and although they are commensurate with the burden they should be supported by demonstrating either cost effectiveness or improved data quality.

The NFS Pilot asks more of respondents than most federally funded studies. It requires participation for all household members for a full week and requests that they track food acquisitions throughout this period. In addition, though the Pilot Study is being implemented several years after FoodAPS-1, we ignored inflation and designed the incentive scheme to be comparable to the incentive scheme used in FoodAPS-1 in terms of total dollar value, as show in the table below. For FoodAPS-1, the data collection agency had to manually enter food or drink items reported on paper diaries into a database after the data collection was over. By contrast, the Pilot study removes the need for this post-survey manual data entry process; as most food items in the Pilot Study are scanned or entered directly into a web-based system. As a result, a slightly higher incentive in the Pilot study is necessary and would still be cost-effective considering that there is no post data collection data entry and no interviewer time involved in conducting telephone interviews during the data collection week.

	NSF Pilot	FoodAPS-1
Incentive structure
Household level	$55 ($50 for completing the Initial Interview and completing the training, $5 for completing the income worksheet)	$100 (for completing both the Initial and Final Interview, returning food books)
Person level	-$3 per person per day (a maximum of $21 per person for completing Food Log) -$50 bonus if household completes Final Interview and all household members complete Food Log)	-$10 for children 11 to 14 (for completing Food Book) -$20 for household members over age 14 (for completing Food Book) -$30 (for initiating the 3 telephone calls)
Maximum incentives
Single-person households	$126	$130
Two-person households	$147	$150 (if both are over age 14) $140 (if one adult & one child between 11 & 14) $130 (if one adult & one child less than 11)
Three-person households	$168	$170 (if all three are over age 14) $160 (if two are over age 14 & one between 11 & 14) $150 (if one is over age 14 & two are between 11 & 14)
Four-person households	$189	$190 (if all four are over age 14) $180 (if three are over age 14 & one between 11 & 14) $170 (if two are over age 14 & two are between 11 & 14)