18. 12-Month Incentive Experiment Findings (Appendix J)

APPENDIX J:
12-MONTH INCENTIVE EXPERIMENT FINDINGS

This page left intentionally blank for double-sided copying.

MEMORANDUM

TO:

Cynthia Miller and Eileen Pederson

FROM:

Jillian Stein and Lisa Schwartz

SUBJECT:

Incentive Experiment Findings

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone (609) 799-3535
Fax (609) 799-0005
www.mathematica-mpr.com

DATE: 11/25/2013
YBP-154

This memo provides a brief overview of the purpose and implementation of the incentive
experiment, and presents the main findings and our recommendations for the 30-month followup survey data collection.
Purpose And Implementation
An incentive experiment was incorporated into the 12-month follow-up survey in order to
test whether offering a higher incentive to youth (1) produced efficiencies by encouraging youth
to complete their surveys early, and (2) improved data quality by increasing response among
otherwise underrepresented segments of the sample.
To answer these questions, respondents were randomly assigned to treatment and control
incentive-experiment conditions. 1 The treatment group received a letter offering them a $40 gift
card if they completed within the first four weeks of data collection. The control group received
a letter offering them a $25 gift card for completing the survey regardless of when they
completed.
Main Findings And Recommendations: 2
The findings show that the $40 incentive condition is associated with 1) greater odds of
completing early; 2) reduced costs due to fewer cases being sent to the phones and the field; and
3) potentially greater representativeness among respondents (although these findings were not
statistically significant). As a result, we recommend continuing to offer the “early bird special”
during the 30-month follow up survey data collection.

1
As a first step in the analysis, we determined that there were no significant differences between the incentivetreatment and control groups, establishing that random assignment worked.
2

For a more detailed review of the results, please see the Detailed Results section on page 2.

An Affirmative Action/Equal Opportunity Employer

MEMO TO: Cynthia Miller and Eileen Pederson
FROM:
Jillian Stein and Lisa Schwartz
DATE:
11/25/2013
PAGE:
2
DETAILED RESULTS, ORGANIZED BY RESEARCH QUESTION
Research Questions:
I.

Were the incentive treatment and control groups roughly comparable at baseline
across key demograhic variables? In other words, did random assignment into the
incentive experiment work correctly?
 Result. There were no significant differences between the incentive
experiment treatment and control groups; random assignment in the incentive
experiment worked correctly.

II.

Within the analytic sample 3, were the incentive treatment and control groups roughly
comparable across key demograhic variables at baseline?
 Results: There were no significant differences between the characteristics of
the incentive experiment treatment and control groups within the analytic
sample.

III.

Within the analytic sample, was the incentive-treatment condition associated with
higher odds of completing the 12 month follow-up survey within the first four weeks
of data collection? If yes, what happens if you add controls for other variables
thought to impact response.
 Result: Those who were offered the $40 incentive had 38 percent higher odds
of completing their survey within the first four weeks, compared to those who
were offered the $25 incentive. This finding remained significant after
controlling for a host of demographic characteristics associated with nonresponse such as gender, age, and race (OR= 1.38 p<.01).

IV.

Did the incentive treatment condition generate efficiencies by encouraging youth to
complete in the first four weeks, hence resulting in fewer cases being sent to the
phone or field?
 Result: As noted above, sample members in the incentive-treatment group
were 38 percent more likely to complete their survey within the first four
weeks of data collection. These cases were not subject to more labor intensive
and costly data collection efforts including contacts from telephone
interviewers, extensive in-house locating, or ultimately field locating.

3
We are still in the process of collecting 12 month survey data, therefore we restricted the analytical sample to
cases that were released at least four-weeks prior to the date that the interim data file was pulled (N=2,562).

MEMO TO: Cynthia Miller and Eileen Pederson
FROM:
Jillian Stein and Lisa Schwartz
DATE:
11/25/2013
PAGE:
3
V.

Was the incentive condition particularly effective for certain segments of the
YouthBuild sample?
 Result: We found that the $40 incentive was associated with higher response
among males, African-Americans, and those who were eighteen or younger,
although these findings were not significantly different. Nevertheless, we plan
to test this further in subsequent analyses including a larger analytic sample.

cc: Lisbeth Goble, Sean Harrington, Cathy Lu, Dan O'Conner