Request for MIHOPE Incentive Increase

TO: Josh Brammer, Office of Information and Regulatory Affairs; Office of Management and Budget

FROM: Nancy Margie, Office of Planning, Research and Evaluation; Administration for Children and Families

RE: Request for Incentive Increase for Family Data Collection in MIHOPE

DATE: December 2, 2014

We are requesting to increase the token of appreciation for families that participate in the MIHOPE follow-up data collection. Our current response rates are much lower than anticipated, and we believe that increasing the tokens of appreciation will in part help to increase the response rates and ensure that we have as complete a follow-up sample as possible. This will also help reduce the possibility of response bias by ensuring that respondents are as similar as possible to the full study sample and that follow-up data collection is equally successful for the study’s program and comparison groups.

Currently, as approved by OMB, MIHOPE follow-up data collection with families (OMB Control No: 0970-0402) provides mothers with a $25 gift card for completing a one-hour survey and an additional $20 gift card for the mother and a $5 board book for the child for completing an additional 1.5 hour data collection in the home. We are requesting to increase these amounts to $40 for completing the survey and $50 plus the board book for completing in-home data collection.

MIHOPE follow-up data collection so far

In MIHOPE, participating mothers are being asked to complete a one-hour survey by telephone when the child is 15 months old. They are also being asked to allow additional data to be collected in their home around the same time. In-home data collection includes the following:

1. Video-recording an interaction between the mother and child using the “three bags” and “clean-up” tasks.

2. Administering the Preschool Language Scales, Fifth Edition (PLS-5), Auditory Comprehension scale to assess the child’s ability to be attentive and respond to stimuli in the environment and to comprehend basic vocabulary or gestures at 15 months.

3. Measuring the child’s weight and height to provide information on whether the child’s growth is within a normal range or exhibits early signs of underweight or obesity. In addition, measure the mother’s weight to assess the effects of home visiting on maternal weight and obesity.

4. Administering the Home Observation for Measuring the Environment (HOME) assessment of the quality and amount of stimulation that the child receives in the home as well as observations of the home environment.

Our target completion rates are 85% for both the survey and the in-home data collection. Unfortunately, early response rates are far below these levels. As of November 19, 64% of 102 cases fielded in April and May 2014 had responded to the follow-up survey and 52% had provided data through the in-home assessments. While the study is still trying to collect data from these families, the data collection effort for those cohorts has gone beyond the planned fielding period.

In addition, the percentage of mothers requiring in-person location has been much higher than expected. Initially, we planned to try to reach mothers by phone for one month and then, if they didn’t respond, use field staff to try to locate them for several additional months. The project assumed that about half of mothers would complete the survey by responding to telephone calls or by calling the Mathematica Survey Operations Center and the remainder would require in-person location. However, only approximately 30% of mothers have completed the survey via phone, requiring field staff to locate and facilitate survey completion for the remainder.

Because it is taking longer than expected to find families, and because more families are requiring in-person location than anticipated, this process is requiring more field staff effort over a longer period of time than originally anticipated, which is resulting in increased data collection costs. Specifically, it is taking almost twice as long to complete a case as originally anticipated (9 hours on average compared to 5 hours expected). Low response rates also reduce the statistical power of the study and may reduce the generalizability of study findings if respondents differ substantially from the full study sample.

The team has taken several actions recently to try to increase the response rates:

1. As previously approved by OMB, MIHOPE is using electronic searching and databases to locate participants. To improve tracking of families, the study team recently added Facebook to the list of electronic searching methods being used. In particular, the study team is now searching for study participants via Facebook using the email address they provided at baseline.

2. Providing more direct oversight and guidance to field staff on how to effectively complete cases.

3. Recruiting and training an additional team of seven field locators to assist in locating hard-to-reach families.

Since mid-August, when the earliest of these changes began, there has been an increase of ten or more percentage points in many of the oldest cohorts. However, we do not expect these changes to bring response rates from the current level up to the desired 85%. Therefore, we are requesting an increase in what families receive for providing follow-up data. Specifically, we are requesting approval to increase the amount provided families for completing the one-hour survey from $25 to $40 and from $20 to $50 (plus the board book for the child worth approximately $5) for completing in-home data collection.

Rationale for increasing incentive amount

The use of monetary incentives in MIHOPE is guided by research that finds that survey participation is a function of positive and negative reasons to participate, as well as trust^1,2,3 This research suggests that incentives can increase a respondent’s willingness to participate by increasing the reward from doing so. Based on this research, monetary incentives have been used for years to increase survey response rates and reduce nonresponse. Many studies document the positive effects of monetary incentives with respect to response rates on mail surveys (Church, 1993; Fox, Crask & Kim, 1988; Harvey, 1987; Hopkins & Gullickson, 1992; Yammarino, Skinner & Childers, 1991). In addition, a growing body of research documents a similar trend on telephone and in-person surveys (Benus and Ackerman, 1971; Gunn and Rhodes, 1981; Kerachsky and Mallar, 1981; Singer, 1999; Webber et al, 1982). See Singer and Ye (2013) for a review of this literature.⁴

In addition to improving response rates, several studies have found that higher incentives can also reduce the level of effort and cost needed to obtain survey responses (Painter et al. 2003;⁵ Singer et al. 1999;⁶ Kovac and Markesich 2002, 2003⁷). For example, Kovac and Markesich at Mathematica conducted two incentive experiments with low income populations to ascertain which incentive amount yielded the best response. One experiment found that a $35 produced a significantly higher response than a $20 incentive. The larger incentive also reduced the time and effort needed to obtain responses. A second experiment showed that a higher incentive for completing the survey in a short time period – $35 if survey completed within 3 weeks of receiving the letter, otherwise $20 – significantly improved response rates. These surveys were shorter in length and less burdensome for respondents than the approved one-hour parent instrument for MIHOPE.

As suggested by this evidence, the cost of increasing the incentives will be at least partly offset by a reduction in the level of effort required to complete a case. As mentioned previously, it is currently taking an average of 9 hours to complete each case, compared to the 5 hours the study assumed it would take. A reduction of 2 hours needed to complete a case (that is, a drop from 9 hours to 7 hours) would offset the cost of the higher incentive, since 2 hours of field staff labor is approximately $40.

Justification for the specific amount

The specific amount that is being requested – $40 for completing a one-hour telephone survey and $50 plus a $5 book board for completing in-home data collection – is consistent with the amount that has been used in a number of recent studies that have successfully achieved good response rates. Here are several examples from MDRC’s work:

• In the Opportunity New York City Family Rewards project, incentives for completing follow-up surveys were increased from $30 to $60. Response rates increased from 64 percent under the smaller incentive to 80 percent with the larger incentive.

• In the SaveUSA study, incentives for completing a follow-up survey were increased from $25 to $50. Response rates increased from 62 percent under the smaller incentive to 80 percent with the larger incentive.

• In the WorkAdvance study in New York City, field locators reported that an incentive of $25 was too low to motivate respondents. The study then increased the incentive to $40. Average response rates in one month of phone interviewing in this study increased by seven percentage points after the incentive was increased. A later increase to $65 increased response rates by an additional 19 percentage points.

This amount is also consistent with recent data collection efforts approved by OMB. Here are several examples:

• In the Supporting Healthy Marriage evaluation conducted for OPRE by MDRC, OMB approved an increase in the incentive from $30 to $50 for adults completing a follow-up survey. Likewise, OMB approved an increase from $30 to $50 per adult for taking part in video-recorded couple interactions. For both parts of follow-up data collection, response rates increased substantially after incentives were increased.

• OMB approved incentives of $35 to $40 for participant surveys in YouthBuild, FACES, and Baby FACES (where surveys were expected to take 45 minutes to an hour) and $35 for in-home assessments in FACES (with a 45-minute in-home assessment) and Baby FACES (with a 2-hour in-home assessment). By comparison, the MIHOPE follow-up survey is expected to take one hour and in-home data collection 1.5 hours.

• In the Subsidized and Transitional Employment Demonstration (STED) and the Enhanced Transitional Jobs Demonstration (ETJD), OMB recently approved an increase from $40 to $50 for a 30-month follow-up survey.

Proposed experiment with pre-payments

The American Association for Public Opinion Research’s Task Force on Survey Refusals notes there is substantial evidence that response rates are higher when incentives are provided with an initial survey request or advance letter.⁸ However, we are not aware of a study that has used pre-payment with a low-income mobile population such as the one involved in MIHOPE. We propose, as part of the MIHOPE study, an experiment to test the effects of providing a small advance payment to study participants.

In particular, 400 study respondents would be randomly divided. One half would receive a $10 gift card with an advance letter notifying them of the data collection effort and an additional $30 gift card after they complete the survey. The other half would receive a $40 gift card after they have completed the survey. Both groups would receive a $50 gift card (if approved by OMB) after they complete in-home data collection.

This test would have a minimum detectable effect of about 10 percentage points (for example, it would be designed to detect an increase from 70 percent under the current payment plan to 80 percent with a $10 pre-payment). If results with the group of 400 showed statistically significant increases in the response rate, we would propose using pre-payments with the remainder of the sample.

1 Advances in Telephone Survey Methodology, Edited by James M, Lepkowski, Clyde Tucker, J. Michael Brick, Edith de Leeuw, Lilli Japec, Paul J. Lavrakas Michael W. Link, and Roberta L. Sangter Copyright 2008 John Wiley & Sons, Inc.

2 Dillman, D. A. (2000) Mail and Internet Surveys: The Tailored Design Method (2^nd Ed.). New York John Wiley & Sons, Inc.

3 Groves, R. M., Singer E., & Corning A. (2000). A leverage-saliency theory of survey participation: description and illustration. Public Opinion Quarterly, 64, 299-308.

4 Singer E. and C. Ye. (2013). “The Use and Effects of Incentives in Surveys.” The ANNALS of the American Academy of Political and Social Science. 645 (112).

5 Painter, D., Chromy, J. R., Meyer, M., Granger, R. A., & Clarke, A. (2003). “Effects of incentives on data collection: a record of calls analysis of national survey on drug use and health.” Proceedings of the Survey Research Methods Section of the American Association for Public Opinion Research, 170-176.

6 Singer, Eleanor, Nancy Gebler, Trivellore Raghunathan, John Van Hoewyk, and Katherine McGonagle.1999. “The effect of incentives in interviewer-mediated surveys.” Journal of Official Statistics 15 (2): 217–30.

7 Kovac M. and J. Markesich. 2002. “Tiered Incentive Payments: Getting the Most Bang for the Buck.” Unpublished manuscript presented at the annual conference of the American Association of Public Opinion Research.

Kovac M. and J. Markesich. 2003. “The Effects of Differential Incentives on Completion Rates: A Telephone Survey Experiment with Low-Income Respondents.” Unpublished manuscript presented at the annual conference of the American Association of Public Opinion Research.

8 AAPOR. 2014. “Current Knowledge and Considerations Regarding Survey Refusals.” To be specific, the report cites the following evidence: “A number of meta analyses have shown that incentives are most effective at increasing response rates when they are non-contingent in nature and thus are provided with the initial survey request in a mail survey (Church, 1993; Singer et al., 2000; Fox et al., 1988) or with an advance pre-notification letter in a telephone survey (Camburn, et al. 1996; Shuttles and Lavrakas, 2004) or in an in-person survey; or given by the interviewer (Berlin et al., 1992; McGrath, 2006; Eyerman et al.,2005). Recently, several studies have shown the effectiveness of a prepaid incentive reducing implicit refusals and thereby in increasing response via the internet (Millar and Dillman, 2011; Messar and Dillman, 2011). Generally, these studies have mailed the incentive to the respondent with a letter that provided information on completing the survey over the internet. Offering an incentive unconditionally decreases refusal rates and increases response rates more so than a comparatively larger contingent incentive, in which the person must respond to obtain the incentive.”

3