Memo 3

Racial and Ethnic Approaches to Community Health across the U.S.

(REACH US) Evaluation

Incentive Experiments

As an Address Based Sampling (ABS) survey, REACH US will attempt to interview respondents via a variety of modes. REACH US will first attempt to obtain a telephone number for the household in order to contact and complete the REACH US instrument via computer-assisted telephone interview (CATI). If (i) we do not obtain a phone number for the address, (ii) we are unsuccessful at contacting the phone number, or (iii) we fail to complete the interview via the telephone, we will attempt to complete the interview via a self administered mail questionnaire (SAQ). If we are still unsuccessful at completing the case, we will attempt to complete the case via face-to-face interviewing.

We have two objectives in testing the use of incentives: (i) providing an estimate of nonresponse bias; and (ii) assessing the cost/benefit of using incentives in general in the study. Given the multiple modes of contact REACH US will attempt, we plan two experiments involving monetary incentives. Given the nature of the questions to be asked on REACH US (general health questions), we do not believe there is a relevant nonmonetary incentive for this study.

First, REACH US plans to offer an incentive to respondents who are mailed a Self-Administered Questionnaire (SAQ) study booklet. We will not have made any contact with those sample members before then. The purpose of this experiment is to establish whether incentives will increase the response rate for the SAQ approach in REACH US; and whether the cost of the increase (if an increase is obtained) can be justified on cost/benefit grounds.

Mail surveys are more burdensome on respondents than interviewer-administered questionnaires; they also tend to suffer relatively low response rates. Mail surveys require additional motivation and effort on the part of the respondent in order to comprehend, complete, and return the instrument¹. Therefore, in order to achieve acceptable response rates from mailed SAQs, we believe it is important to explore the use of incentives for those asked to complete the survey via mail SAQ.

Half the sample will be randomly flagged to receive the incentive while the remaining sample will be used as a control group. Those in the experimental treatment group who are mailed an SAQ will also receive a $5 incentive with the packet; within the experimental group half of the households will receive a promise of an additional $10 upon return of the completed SAQ. The token of appreciation will be given to the REACH US household and not individual respondents to avoid encouraging the completion of questionnaires for ineligible respondents.

Second, REACH US plans to conduct an experiment with an incentive to telephone respondents after a second CATI refusal. Cases eligible for this treatment would be cases that have completed the household screener, are known to meet the REACH US interview eligibility criteria for the community, and since being selected for the interview have twice refused to complete the interview (e.g., refused by saying “not interested,” “don’t have time,” etc.). These respondents are extremely important to the success of the survey because they represent the target racial or ethnic population for a given community, and the overall eligibility rate with in the community may be quite low.

The purpose of following up those telephone refusers is to assess the potential nonresponse bias that would be caused by failing to recruit those sample members into the respondent group by phone. The purpose of the experiment is to explore whether the refusal conversion letter deals adequately with the potential bias, or whether those still resistant area different in significant ways from the others. The experiment is tightly focused on those known to be eligible.

Half of the sample in each REACH community will randomly be flagged to be eligible for an incentive (should they refuse twice in CATI) while the remaining sample will be used as a control group. In the experimental group, after the second CATI refusal, a refusal conversion letter that addresses the respondent’s concerns and $5 will be mailed to the CATI respondent with a promise of an additional $10 token of appreciation upon returning the completed questionnaire (control group refusals will receive a conversion letter only). If the respondent refuses again after receiving the refusal conversion letter mailing, the contractor will cease all telephone contact with the respondent. Control group respondents who complete the interview will not receive an incentive. The first incentive experiment described above will inform the treatment to be used in nonresponse bias studies in later years, in particular in terms of the desirability of including the promise of additional payment.

The REACH contractor has substantial experience with the impact of offering incentives. For example, in several experiments conducted on the National Immunization Survey, which, like REACH US has low survey eligibility, incentives were consistently found to increase rates, reduce costs, and lower burden².

These experiments are motivated by a number of factors. First, ABS surveys are relatively new and thus little is known about the interaction of the methodology with established incentive methodology. REACH, by design, is a multimode survey and thus is it is important to determine how incentives interact with survey response across modes, especially those (telephone and mail) where there has been historical concern about response rates. Many previous studies have confirmed the effectiveness of incentives for increasing response across a variety of modes³, but none have directly explored the effectiveness of incentives on an ABS design. NORC, the contractor selected to conduct the REACH interview, and several REACH project members from NORC, have broad experience conducting experimentation and implementing incentive protocols.  NORC has found the incentives to be successful at increasing response on studies like REACH and generally find that incentives reduce survey cost and burden on any study with a relatively low eligibility. 

The objective of REACH US is to measure health outcomes for minority and underserved populations. The study populations are relatively rare (expected study eligibility at some sites is less than 10% of the population). Thus, it is crucial to the success of the survey that these populations fully participate. The incentive experiment will allow us to both increase participation and measure the extent to which these underserved populations require incentives to encourage participation. Further, evidence for prior rounds of the REACH 2010 survey suggests that the study populations and/or similar populations are extremely difficult to reach. Incentives will thus increase our response rate and data quality.

We believe the use of incentives will reduce overall burden of this survey. Given the relatively low survey eligibility in some study areas, each completed interview requires a substantial number of contacts to ineligible households. By offering incentives to potential respondents, we increase the likelihood that a eligible household will complete the survey and thus decrease the overall number of households we will need to contact. Our experimental treatments will allow us to test this hypothesis

Finally, we believe the use of incentives will reduce overall data collection costs. Again, given the relatively low eligibility of some study areas, each completed interview requires a substantial number of contacts, each with an associated cost. Particularly in the telephone treatment, where the household receiving the incentive will be known to be eligible after extensive screening, converting a refusal may save us substantial data collection effort. Our experiment will allow us to test this.

The treatment (incentive) and control (non-incentive) groups will be compared on a variety of characteristics:

• Response, refusal, and conversion rates: Do incentives increase cooperation?

• Key questionnaire measures: Do respondents who complete the survey after receiving an incentive differ (demographically, socioeconomically, on key survey measures of health, etc.) from those in the same category who complete the survey without an incentive?

• We plan to complete face-to-face interviews with mail/phone non-responders from all categories (including those offered and not offered an incentive); thus we can examine the characteristics of original non-responders in the two groups to examine the effectiveness of the incentive approach. This provides an unusual opportunity to validate the results of the experiments.

• Given the results of the control and treatment groups, what is the expected cost reduction or increase of implementing the incentive treatment for all cases versus offering incentives to no cases? Using rate and cost information from the control and treatment group, we will be able to calculate the total cost of the survey had the incentive treatment been applied to all cases versus applied to none.

• Given the results of the control and treatment groups, what is the expected increase or decrease in survey burden of implementing the incentive treatment for all cases versus offering incentives to no cases? Using rate information from the control and treatment group, we will be able to calculate the total sample we would need had the incentive treatment been applied to all cases versus applied to none.

1Church, 1993, “Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis,” Public Opinion Quarterly, 57:62-79

2 See, for example Brooks, Keeshawna, Martin Barron, Margrethe Montgomery, Ben Skalland, and James Singleton. 2007. “The impact of incentives on representativeness in the National Immunization Survey.” Presented at the Annual Meeting of the American Association for Public Opinion Research, Anaheim, CA.

3 For a review of the impact of incentives on mail surveys, see Church ibid. . For telephone and face-to-face surveys, see Singer, Van Hoewyk, Gebler, Raghunathan, McGonagle, 1999, “The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys,” Journal of Official Statistics, 15:217-230.

4