N. Incentives and Response Rates_2022 09 02

Appendix N. Incentives and Response Rates

This page has been left blank for double-sided copying.

FNS is requesting a post-survey incentive of a $30 Visa gift card for sample members completing the participant survey and a $50 Visa gift card for participant focus group and participant in-depth interview (IDI) respondents. Research consistently shows that monetary incentives increase response rates. Incentives also reduce survey non-response bias without compromising data quality.^1,2,3 Post-pay incentives encourage participants to stay engaged with the survey or data collection activity until they have completed it and help offset expenses and time associated with participation. Below, we describe the literature that motivated the proposed post-paid incentive approach for the survey and the focus group and IDIs.

Research indicates that offering monetary incentives both improves response rates and mitigates nonresponse bias across different respondent populations, particularly among low-income respondents, those residing in rural areas, and those receiving federal nutrition assistance benefits.^2,4 On the Project LAUNCH Cross-Site Evaluation (OMB number 0970-0373, expired October 31, 2019), the study initially did not offer an incentive to parents invited to complete a web-based survey. The team found that these early respondents were not representative of their communities. Specifically, minorities, individuals with lower incomes and education levels, and those who worked part-time or were unemployed were underrepresented. After introducing a $25 post-pay incentive, the study saw increased completion rates and sample representativeness.⁵

Research has also shown a positive relationship between higher incentives and increased survey response rates. A meta-analysis by Singer et al. found response rates increased by approximately one-third of a percentage point, on average, with every one dollar increase in incentive amount.² Experiments conducted by the U.S. Census Bureau with the Survey of Income and Program Participation (SIPP) in 2014 compared the effects of $10, $20, and $40 post-pay incentives on survey response. A $40 post-pay incentive increased response rates by 3.0% compared to the $0 control group, while a $20 incentive increased response rates by 1.1%.⁶ The $40 incentive also increased response rates among subgroups of low-income respondents.

While higher incentives are associated with higher response rates, research suggests that the marginal increase in response rate eventually decreases as incentive amounts increase. A meta-analysis by Mercer et al. saw improvements to response rates in telephone administered surveys up to an incentive amount of $30.⁷ Offering higher incentives may still result in increased survey response rates, but at a larger cost with less benefit. This same meta-analysis concluded that while survey burden had a negative effect on overall response rate, survey burden did not significantly impact the relationship between incentive amount and response rate, a finding also noted by Singer et al.² This finding suggests that survey incentive effectiveness is more strongly associated with factors like survey mode and population rather than survey burden.

FNS proposes to provide participant survey respondents with a $30 Visa gift card as a post-incentive upon completion of the survey. We expect $30 to be a cost-effective way to reduce survey nonresponse bias and increase response rates among our population of SNAP participants. In order to improve representativeness and response rate, and minimize the cost of replacing “no-show” respondents, FNS plans to offer a $50 Visa gift card for the 90-minute focus group and IDIs. Given the greater burden placed on respondents for these qualitative data collection activities, a larger incentive is necessary to obtain information from a diverse set of respondents. The $50 amount is consistent with the literature⁸ as well as other OMB-approved information collections. Together, this incentive strategy aims to increase sample representativeness and help offset expenses and time associated with participation in data collection activities.

1 Singer E, Ye C. The use and effects of incentives in surveys. The ANNALS of the American Academy of Political and Social Science. 2013;645(1):112–41.

2 Singer, Eleanor, et al. "The effect of incentives on response rates in interviewer-mediated surveys." Journal of official statistics 15.2 (1999): 217.

3 Singer, E., and R.A. Kulka. “Paying Respondents for Survey Participation.” In Studies of Welfare Populations: Data Collection and Research Issues. Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, edited by Michele Ver Ploeg, Robert A. Moffitt, and Constance F. Citro. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press, 2002, pp. 105–128.

4 Bonevski, Billie, et al. "Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups." BMC medical research methodology 14.1 (2014): 1-29.

5 LeFauve, K., K. Rowan, K. Koepp, and G. Lawrence. “Effect of Incentives on Reducing Response Bias in a Web Survey of Parents.” Presented at the American Association of Public Opinion Research Annual Conference, Denver, CO, May 16–19, 2018.

6 Westra, Ashley, Mahdi Sundukchi, and Tracy Mattingly. "Designing a multipurpose longitudinal incentives experiment for the Survey of Income and Program Participation." Proceedings of the 2015 Federal Committee on Statistical Methodology (FCSM) Research Conference, available at https://www.reginfo.gov/public/do/DownloadDocument?objectID=77072601 (last accessed June 9, 2022). 2015.

7 Mercer, Andrew, et al. "How much gets you how much? Monetary incentives and response rates in household surveys." Public Opinion Quarterly 79.1 (2015): 105-129.

8 Krueger, Richard A. Focus groups: A practical guide for applied research. Sage publications, 2014.