Incentive Lit Review

Attachment 4: Incentives

Research and Evaluation Survey for the Public Education Campaign on Tobacco among LGBT (RESPECT)

Incentive Plan for Research and Evaluation Survey for the Public Education Campaign on Tobacco among LGBT (RESPECT LGBT)

This section presents the rationale for the incentives we propose for participation in RESPECT LGBT surveys. Incentives will be provided to encourage completion of the 5-minute in-person screener survey ($10), the 30-minute baseline (pre-campaign launch) web survey (promised $20 incentive with a $5 “early response” bonus for completing the survey within 2 days of invitation) and the 40-minute follow-up (post-campaign launch) surveys (promised $20 with a $5 “early response” bonus for completing the survey within 2 days of invitation).

The incentive plan for this study is based on the need to ensure cooperation from this hard-to-reach and specific population of young adult (18- to 24-year-old) members of the LGBT community. Proposed incentives are intended to recognize the time burden placed on participants, encourage their cooperation, and convey appreciation for contributing to this important study.

Overarching Finding

Evidence Supports Use of Incentives to Increase Response Rates and Reduce Costs

Studies indicate that incentives significantly improve response rates and retention (Kulka et al. 2005, Singer & Ye 2013, Trussell & Lavrakas 2004). Studies also show that incentives reduce the cost per survey completed through their influence on the amount of effort that is required to achieve a completed interview (Beebe et al. 2005, Kennet et al. 2005). There is little evidence to suggest negative effects of incentives on data quality, sample composition, and response distribution (Singer & Ye 2013). For these reasons and because of the large body of evidence supporting these findings, incentives have been supported in many OMB-approved information collection efforts.

Detailed Findings

Evidence for the Effectiveness of Incentives for In-Person Intercept Surveys

Incentives are particularly important in intercept surveys. By definition, intercept respondents are busy doing something else at the time they are intercepted. In the case of this study, when intercepted, respondents will be entering a bar, leaving the bar to go elsewhere, or are likely to be otherwise engaged in interactions with others inside the bar. Unlike a mail or web survey that can be done at the respondent’s leisure, or a telephone or in-person interview that can be scheduled at the respondent’s convenience, in an intercept study the invitation to participate is a relatively immediate one, and respondents are likely to require motivation to stop what they are doing. While there is little published experimental research that examines the effectiveness of incentives vs. no incentives with intercept surveys, there are numerous examples of public health research that has used cash incentives when intercepting respondents at bars or other “party” venues. Incentives within this literature tend to range from $5-$10 for completing a brief survey when entering the venue (e.g., Bourdeau et al. 2015, Guillory et al. 2015, Miller et al. 2003, Voas et al. 2013) and were typically $20 when both survey data and biological measures were collected upon leaving the venue data (e.g., Bourdeau et al. 2015, Miller et al. 2003, Voas et al. 2013). A meta-analysis of incentive use during intercept studies in the transportation field suggests that incentives that are paid at the time of completion have a larger impact on response rates than promised incentives (Schaller 2005).

Evidence for the Effectiveness of Promised Incentives to Complete Web Surveys

Promised incentives are an especially practical and effective means for maximizing response for web surveys (e.g., Birnholtz et al. 2004, Bosnjak & Tuten 2003, Gajic et al. 2012). Two meta-analyses conducted by Goritz (2006) found that incentives increased the odds that a person would start a web survey by 19 percent and increased the odds that a person would complete a web survey by 27 percent.

Evidence for Effectiveness of Promised “Early Response” Bonus Incentives

An important strategy for cost containment is to encourage participants to respond, and respond quickly, with minimal additional prompting. LeClere et al. (2012) found that respondents who were offered an “early response” bonus were more likely to respond than those not offered an incentive or who were offered a pre-paid incentive. In addition, responders in the “early response” category responded more quickly than responders who were not offered the early response bonus, reducing the cost of follow-up attempts for respondents offered this “early response” bonus. Similarly, Coopersmith et al. (2014) found that promised early response bonus incentives yielded both higher response rates and faster responses to a web survey compared to a promised incentive with no bonus, a pre-paid incentive, or a non-response conversion incentive.

Evidence for Effectiveness of Incentives in Minimizing Attrition in Longitudinal Surveys

Multiple studies indicate that incentives for completion of the baseline survey lowered nonresponse and increased retention over the life of the study (James 1997, Singer et al. 1999, Singer and Kulka 2002, McGrath 2006, Goldenberg et al. 2009). For example, the U.S. Census Bureau has also experimented with and begun offering incentives for several of its longitudinal panel surveys, including the Survey of Income and Program Participation (SIPP) and the Survey of Program Dynamics (SPD). SIPP has conducted several multi-wave incentive studies, most recently with their 2008 panel, comparing results of $10, $20, and $40 incentive amounts to those of the $0 control group. The study has examined response rate outcomes in various subgroups of interest (e.g., the poverty stratum), use of targeted incentives for non-interview cases, and the impact of base wave incentives on participation in later waves of data collection. Overall, the results suggest that $20 incentives increase response rates and also improve the conversion rate for non-interview cases. Similarly, SPD has conducted four incentive studies, testing $20, $40, $50, and $100 amounts in an effort to minimize attrition in subsequent waves of the study. Incentives were found to have a positive impact on both response and attrition rates, most recently, a fourth incentive study found that the average interview rate greatly increased with the use of incentives (Creighton et al. 2007). Similarly, Jäckle and Lynn (2008) found that incentives at multiple waves significantly reduced attrition at all waves.

Evidence for Effectiveness of Incentives in Reducing Cost Per Completed Interview

Incentives, and the amount of incentives, can reduce the cost per case completed through their influence on the amount of effort that is required to achieve a completed interview. Evidence in support of this conclusion is provided by the incentive experiments conducted for the National Survey on Drug Use and Health (NSDUH, Substance Abuse and Mental Health Services Administration). Cost per interview in the $20 group was 5 percent lower than the control (no incentive), and in the $40 group costs were 4 percent lower than the control. The cost savings were gained by interviewers spending less time trying to obtain cooperation from respondents (Kennet et al. 2005). These savings were realized through reduced interviewer labor as well as reduced travel costs (mileage, tolls, parking, etc.). A similar finding was produced by an incentive experiment conducted for the National Survey of Family Growth (NSFG, National Center for Health Statistics) Cycle 5 Pretest which examined $0, $20, and $40 incentive amounts. As in the NSDUH experiments, the additional incentive costs were more than offset by savings in interviewer labor and travel costs (Duffer et al. 1994).

Respondents to a telephone study in the Washington, D.C., area about experience with TANF recipiency received a pre-notification letter containing $2 (Markesich & Kovac 2003). The letter noted that upon completion of the telephone interview, the respondent would receive either another $18 or another $33. While the small sample size and uneven distribution of the experimental conditions prevented the response rates among the two incentive amounts from differing significantly, there was a difference in the level of effort needed to achieve desired response rate targets. Those individuals who were offered $35 for completing an interview achieved a 68.7 percent response rate about 2 ½ weeks faster, and with fewer contact attempts, than did those sample members who were offered $20.

The Mayo Clinic study of Medicaid recipients, previously cited, similarly found that the overall cost per complete of surveys in the incentive group were lower than in the group that did not receive an incentive (Beebe et al. 2005). This group required fewer contacts and fewer expensive telephone follow ups. The Latino subgroup was an exception to this finding, with the cost for completed interviews being more in the incentive group.

Incentive Amounts Offered by Other Federally-Funded Surveys

Many federally-sponsored surveys offer incentives to gain cooperation, and these incentives range from $15 to $125, depending on respondent burden. The National Immunization Survey (NIS, National Center for Immunizations and Respiratory Diseases), for example, offers a combination of $5 pre-paid and $10 promised incentives to encourage eligible nonrespondents to participate. The National Survey of Adoptive Parents of Children with Special Health Care Needs (Department of Health and Human Services) offers parents $25 for participation in a 35-minute telephone survey. Interviewers in the NSDUH currently offer $30 for an interview that averages 60 minutes, and the National Survey of Family Growth (NSFG, Centers for Disease Control and Prevention) offers $40 for interviews that are about 60 minutes for males and 80 minutes for females. Over rounds 1 through 10 of the National Longitudinal Survey of Youth 1997 cohort (NLSY97, Bureau of Labor Statistics), incentives offered to respondents ranged from $10 to $50 in an attempt to minimize attrition across waves of data collection. In order to improve response rates, reduce the number of contacts required to gain cooperation, and address respondent concerns about interview burden, the National Survey of Child and Adolescent Well-Being (NSCAW, Administration for Children and Families) in 2002 doubled the incentive offered to respondents from $25 to $50. The Early Childhood Longitudinal Study-Birth Cohort (ECLS-B, U.S. Department of Education) offered parent participants $50 and a children’s book for the first wave and $30 and a children’s book for subsequent waves of data collection. Incentives on the National Health and Nutrition Examination Survey (NHANES, National Center for Health Statistics) range from $20 to $125 depending on the survey and physical exam components in which respondents agree to participate. The Evaluation of the Public Education Campaign on Teen Tobacco (ExPECTT, Food and Drug Administration) employs a $20 promised incentive for 30 and 45 minute in-person or web surveys. Likewise, the Evaluation of the Fresh Empire Campaign on Tobacco (EFECT, Food and Drug Administration) employs a $25 promised incentive for 30 and 45 minute in-person or web surveys.

Evidence for Effectiveness of Incentives to Recruit LGBT Young Adult Populations for Research Studies

Additional research studies have used similar incentives to the federally-funded studies discussed above to effectively recruit members of the LGBT community, noting that members of the LGBT community are a particularly difficult-to-reach population and that incentives for participating in research are an important component of recruiting research participants (Meyer & Wilson 2009). Jones et al. (2008) provided 18- to 30-year-old black men who have sex with men $20 gift card incentives for participating in cross-sectional surveys. Silvestre et al. (2006) paid minority men who have sex with men $25 for an HIV epidemiological study. Remafedi and Carol (2005) offered LGBT youth $20 incentives and cessation referrals for participating in interviews for designing tobacco prevention campaigns.

References

Beebe, T. J., Davern, M. E., McAlpine, D. D., Call, K. T., & Rockwood, T. H. (2005). Increasing response rates in a survey of Medicaid enrollees: the effect of a prepaid monetary incentive and mixed modes (mail and telephone).Medical Care, 43(4), 411-414.

Birnholtz, J. P., Horn, D. B., Finholt, T. A., & Bae, S. J. (2004). The Effects of Cash, Electronic, and Paper Gift Certificates as Respondent Incentives for a Web-Based Survey of Technologically Sophisticated Respondents. Social Science Computer Review, 22(3), 355-362.

Bourdeau, B., Miller, B., Johnson, M., Voas, R. (2015). Method of transportation and drinking among club patrons. Transportation Research Part F, 32, 11-22.

Bosnjak, M., & Tuten, T. L. (2003). Prepaid and Promised Incentives in Web Surveys: An Experiment. Social Science Computer Review, 21(2), 208-217.

Creighton, K. P., King, K. E., & Martin, E. A. (2007). The use of monetary incentives in Census Bureau longitudinal surveys. Survey Methodology, 2.

Coopersmith, J., Vogel, L. K., Bruursema, T., & Feeney, K. 2014. Effects of Incentive Amount and Type on Web Survey Response Rates. In JSM Proceedings, Survey Research Methods Section. Alexandria, VA: American Statistical Association. 4463-4474.Retrieved from http://www.amstat.org/sections/srms/proceedings/y2014/Files/400288_500797.pdf

Duffer, A., Lessler, J., Weeks, M., & Mosher, W. (1994). Effects of incentive payments on response rates and field costs in a pretest of a national CAPI survey. Paper presented at the 49th annual conference of the American Association for Public Opinion Research, Danvers, MA.

Gajic, A., Cameron, D., & Hurley, J. (2012). The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. The European Journal of Health Economics, 13(6), 789-799.

Göritz, A. S. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1(1), 58-70.

Goldenberg, K. L., McGrath, D., & Tan, L. (2009). The Effects of Incentives on the Consumer Expenditure Interview Survey. In Proceedings of the American Statistical Association Joint Statistical Meetings, Survey Research Methods Section (pp. 5985-5999).

Guillory, J., Johns, M., Farley, S., Ling, P. (2015). Loose Cigarette Purchasing and Nondaily Smoking Among Young Adult Bar Patrons In New York City. American Journal of Public Health. 105(8), e140-e147.

James, D. E. (1997). Environmental Incentives: Australian Experience with Economic Instruments for Environmental Management: Consultancy Report. Community Information Unit, Department of the Environment, Sport and Territories.

Jäckle, A., & Lynn, P. (2007). Respondent incentives in a multi-mode panel survey: cumulative effects on nonresponse and bias. Institute for Social and Economic Research.

Jones, K.T., Gray, P., Whiteside, Y.O., Wang, T., Bost, D., Dunbar, E., Foust, E., Johnson, W.D. (2008). Evaluation of an HIV prevention intervention adapted for black men who have sex with men. American Journal of Public Health, 98(6), 1043-1050.

Kennet, J., Gfroerer, J., Bowman, K. R., Martin, P. C., & Cunningham, D. B. (2005). Introduction of an incentive and its effects on response rates and costs in NSDUH. In Kennet, J., & Gfroerer, J. (Eds.), Evaluating and improving methods used in the National Survey on Drug Abuse (DHHS Publication No. SMA 05-4044, Methodology Series M-5). Rockville MD: Substance Abuse and Mental Health Services Administration, Office of Applied Studies.

Kulka, R. A., Eyerman, J., & McNeeley, M. E. (2005). The use of monetary incentives in federal surveys on substance use and abuse. Journal of Economic & Social Measurement, 30(2/3), 233-249.

LeClere, F., S. Plumme, J. Vanicek, A. Amaya, and K. Carris. (2012). Household early bird incentives: leveraging family influence to improve household response rates. American Statistical Association Joint Statistical Meetings, Section on Survey Research.

Miller, B., Byrnes, H., Branner, A., Johnson, M., Voas, R. (2003). Group Influences on Individuals’ Drinking and Other Drug Use at Clubs. Journal of Study of Alcohol and Drugs, 74(2),280-287.

Markesich, J., & Kovac, M. (2003). The Effects of Differential Incentives on Completion Rates: A Telephone Survey Experiment with Low-Income Respondents. In 58th annual conference of the American Association for Public Opinion Research, Nashville, TN.

McGrath, D. E. (2006, August). An incentives experiment in the US consumer expenditure quarterly survey. In JSM proceedings.

Meyer, I.H., Wilson, P.A. (2009). Sampling lesbian, gay, and bisexual populations. Journal of Counseling Psychology, 56(1), 23-31.

Remafedi, G. & Carol, H. (2005). Preventing tobacco use among Lesbian, Gay, Bisexual, and Transgender Youths. Nicotine & Tobacco Research, 7(2), 249-256.

Schaller, B. (2005). On-Board and Intercept Transit Survey Techniques. Transit Cooperative Research Program (TCRP) Synthesis 63, published by Transportation Research Board, Washington.

Silvestre, A.J., Hylton, J.B., Johnson, L.M., Houston, C., Witt, M., Jacobson, L., Ostrow, D. (2006). Recruiting minority men who have sex with men for HIV research: Results from a 4-city campaign. American Journal of Public Health, 96(6), 1020-1027.

Singer, E., Groves, R. M., & Corning, A. D. (1999). Differential incentives: Beliefs about practices, perceptions of equity, and effects on survey participation. Public Opinion Quarterly, 251-260.

Singer, E., & Kulka, R. A. (2002). Paying respondents for survey participation. Studies of welfare populations: Data collection and research issues, 105-128.

Singer, E., & Ye, C. (2013). The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 112-141.

Trussell, N., & Lavrakas, P. J. (2004). The influence of incremental increases in token cash incentives on mail survey response is there an optimal amount?. Public Opinion Quarterly, 68(3), 349-367.

Voas, R., Johnson, M., Miller, B. (2013). Alcohol and drug use among young adults driving to a drinking location. Drug and Alcohol Dependence, 132, 69-73.

4