Attachment 10 - Results of the Round 10-11 Incentive Experiment

Attachment 10—Results of the Round 10-11 incentive experiment

The NLSY97 is a long-term study in which the same subjects are interviewed annually. For that reason, respondents in all prior rounds have been offered financial and in-kind incentives as a means of securing their long-term cooperation and arresting the decline of response rates.

With OMB approval, the Rounds 10-11 incentive experiment began in December of 2006 after about 5,085 interviews were completed for Round 10. The remaining 3,809 sample members were randomized into three equal-sized groups: the control group, the discretionary in-kind group, and the cash payment group. Siblings received the same group assignment, and if one sibling had completed the interview in Round 10 before the experiment began, he or she was in the same group for Round 11. Here is a summary of what each group received:

(1) The nonexperimental group received the $30 in Rounds 10 and 11 plus any bonuses. This can be regarded as the most cooperative group.

(2) The control group received $30 in cash as well as the usual bonuses for respondents who were returning to the survey ($10 per missed round, up to $30), and this regime continued in Round 11.

(3) The discretionary in-kind treatment group received $30 in cash plus bonuses for missed rounds. In addition they received in-kind payments that averaged $20 in value with a maximum value of $30. Field managers and field interviewers were allowed to determine the form of the in-kind incentive so that they could tailor it to the specific respondent. This continued in Round 11.

(4) The cash treatment group received $50 in cash as well as bonuses for missed rounds. This regime continued in Round 11.

In Round 12, all respondents received the base incentive of $30 in cash and up to $20 worth of in-kind incentives. This was motivated by the success of in-kind incentives in improving response rates in Rounds 10 and 11 as well as the flexibility of in-kind incentives compared to cash.

Results of the incentive experiment from Rounds 10-12

Tables 11.1 and 11.2 show the completion rates in Rounds 10-12 for the different groups in the experiment. The first 5,085 respondents who completed interviews prior to the start of the experiment were not included. To separate the more cooperative and less cooperative sample members within the experiment, the tables also contain response rates for those who participated in the previous round (that is, more cooperative) and those who were not interviewed in the last round (less cooperative).

In Round 10, the control group had significantly lower rates of completion (61.6%) than both the cash and in-kind groups (68.7% and 68.4% respectively). The two experimental groups had comparable rates of completion. In Round 11, these differences persisted. The control group had a significantly lower completion rate (62.5%) than the cash or in-kind incentive groups (71.1 and 69.8%, respectively). In Round 11, cash incentives slightly outperformed in-kind incentives, but the differences were not statistically significant. These differences between the control group and two experimental groups are observable in Tables 11.1 and 11.2 for the respondents who missed the previous round and those who participated in the previous round for both Rounds 10 and 11. In both rounds, the cash group slightly outperformed the in-kind group for prior-round noninterview respondents.

Another interesting fact that emerges from the Tables 11.1 and 11.2 is that the noninterviews in Round 10 returned to the survey in Round 11 at a slower rate than Round 9 noninterviews in Round 10. Respondents who were not interviewed in Round 9 were offered at least twice as much in Round 10 as they had been in Round 9. Among treatment group respondents, the value could be tripled. In contrast, Round 10 noninterview respondents were generally offered more modest increases of $10 over their most recent (refused) offer. Thus, slower noninterview returns in Round 11 as compared to Round 10 are consistent with our expectation that the total incentive amount offered would affect cooperation.

Table 11.1: Final Round 10 Status by Incentive Experiment Group and Final Round 9 Status
	Full Experiment					Control Group					In-Kind Group					Cash Group
	All		Completes			All		Completes			All		Completes			All		Completes
	N		N	%		N		N	%		N		N	%		N		N	%
Total	3809		2521	66.2%		1288		793	61.6%		1260		862	68.4%		1261		866	68.7%
Round 9 Comp	2382		2039	85.6%		782		644	82.4%		806		703	87.2%		794		692	87.2%
Round 9 NIR	1427		482	33.8%		506		149	29.4%		454		159	35.0%		467		174	37.3%

Table 11.2: Final Round 11 Status by Incentive Experiment Group and Final Round 10 Status
	Full Experiment					Control Group					In-Kind Group					Cash Group
	All		Completes			All		Completes			All		Completes			All		Completes
	N		N	%		N		N	%		N		N	%		N		N	%
Total	3809		2580	67.7%		1288		805	62.5%		1260		879	69.8%		1261		896	71.1%
Round 10 Comp	2521		2242	88.9%		793		686	86.5%		862		774	89.8%		866		782	90.3%
Round 10 NIR	1288		338	26.2%		495		119	24.0%		398		105	26.4%		395		114	28.9%

Table 11.3: Final Round 12 Status by Incentive Experiment Group and Final Round 11 Status
	Full Experiment					Control Group					In-Kind Group					Cash Group
	All		Completes			All		Completes			All		Completes			All		Completes
	N		N	%		N		N	%		N		N	%		N		N	%
Total	3809		2651	69.6%		1288		872	67.7%		1260		892	70.8%		1261		887	70.3%
Round 11 Comp	2574		2349	91.3%		803		729	90.8%		877		810	92.4%		894		810	90.6%
Round 11 NIR	1235		302	24.5%		485		143	29.5%		383		82	21.4%		367		77	20.9%

In Table 11.3 we still see differences in completion rates between the control group (67.7%) and the experimental groups (70.8% for the in- kind and 70.3% for the cash group) overall. The completion rate is higher for the Round 10-11 treatment groups compared to the control group in the experiment as a whole and is higher for the in-kind group compared to the control and cash groups for the respondents who completed Round 11. However, for prior round noninterviews the completion rates were smaller for both the experimental groups compared to the control group.

This experimental design offered the NLS program an opportunity to test the effectiveness of discretionary in-kind incentives and large increases in cash incentives. Results from rounds 10 and 11 indicate that response rates have increased for respondents in both the two treatment groups. Our results also suggest that an in-kind incentive supplement makes the best use of project dollars to improve respondent cooperation, invest in the unique relationship that each respondent has with the NLSY97, and permit some flexibility in aggregate cost. In Round 12, where everyone was eligible for the same incentive, we still see small differences between the completion rates for the Round 10-11 control groups and the groups that received a larger amount in the prior round, indicating that there may be some goodwill created in the mind of respondents due to the large incentive increase in Rounds 10-11. When data from Round 13 become available, we can look further into the long-term effects of incentives as well as judge whether the additional in-kind payments had an impact on completion rates or how quickly the cases were completed during the field period.