Attachment 9 - Preliminary Summary of Rounds 10-11 Incentive Experiment

Attachment 9—Preliminary Summary of Rounds 10-11 Incentive Experiment

Design

In October, 2006, BLS proposed conducting an experiment in Rounds 10 and 11 to study two questions that would help us craft an effective long-term incentive policy:

1. Can targeted in-kind incentives result in higher response than cash incentives?

2. Will large increases in incentives have significantly greater impact than smaller increases in the current round and in subsequent rounds of data collection?

With OMB approval, the Rounds 10-11 incentive experiment began in December of 2006. At that time, interviews had been completed with 5,000 respondents who can be regarded as very cooperative. The remaining 3,825 sample members were randomized into three equal sized groups: the control group, the discretionary in-kind treatment group, and the cash payment group. This randomization occurred across families so that siblings would be treated equally. If a sample member in the experiment has a sibling who already had completed a Round 10 interview before the experiment began, the siblings are in the same group for Round 11.

Control group: The control group continued to receive the basic Round 10 respondent incentive of $30 in cash. In addition, respondents who had missed previous rounds continued to receive the increased payments previously approved ($10 per missed round, up to $30). This regime continues for Round 11 as well.

Discretionary in-kind treatment group: This first treatment group received the previously approved $30, and respondents who had missed previous rounds were eligible for the increased payments of $10 per missed round. Respondents in this group were also eligible to receive in-kind payments that averaged $20 in value with a maximum value of $30. All respondents in this treatment group received some form of in-kind incentive. BLS allowed field managers and field interviewers the ability to determine the in-kind incentive that they judged would be most effective at securing cooperation. This regime continues for Round 11 as well.

Cash payment treatment group: For the second treatment group, BLS proposed increasing the respondent payment to $50 in cash. As with the control group and the in-kind treatment group, respondents in the cash treatment group who had missed previous rounds received the payments of $10 per missed round (up to $30 maximum). This regime continues for Round 11 as well.

The purpose of this large increase in payments – from Round 9 to Round 10, this represents a $30 increase in respondent incentives – was to examine whether such a large increase would have any impact on response rates. Because the respondents who had already completed the survey at this point could not be affected by the increased payments, we effectively stratified our respondents by their willingness to participate. The failure to detect an impact of the increased respondent incentives for this group of sample members would provide strong evidence that respondent incentives are not effective at increasing response rates.

Remaining respondents: In Round 11, respondents who had completed their survey prior to random assignment and who are not siblings of participants included in the experiment continue to receive the $30 respondent payment authorized for Round 10.

This experimental design offers the NLS program an opportunity to test the effectiveness of discretionary in-kind incentives and large increases in cash incentives. Although it is early, there are some statistically significant results to show that response rates have increased for respondents in either of the two treatment groups. Running the experiment for two years also affords us the opportunity to look at the dynamic responses to the increased incentive. Thus, we can learn how sample members react in the second year. Do respondents come to expect such large increases in the incentive? Do any first-year improvements in response rates persist into the second year (as our previous results suggest)? Does the increase in the incentive encourage previously reluctant sample members to become more cooperative? Do they complete an interview earlier in the fielding period?

Preliminary Results

For Round 10, in-kind payments averaged just over $19. The most common in-kind gift was some type of gift card, varying somewhat in value and extensively in the choice of merchant. Merchants whose gift cards were chosen most often included discount stores, gas stations, retail electronics stores, coffee shop chains, video and entertainment rental, and grocery stores. We see within-interviewer variation in gift cards, indicating that interviewers are selecting different cards for different respondents. Other in-kind gifts included refreshments to be consumed during or immediately after the interview (for example, coffee during the interview or a pizza that the interviewer brought to the respondent’s home), toys and other gifts for respondent’s children, and other tokens, including a calendar and a pair of pants. We did not collect data on incentives offered (but not paid) to respondents who did not complete the interview.

The top panel in Table 9-1 provides final completion rates for the 3,825 cases in the Round 10 incentive experiment. The two treatment groups completed interviews at a higher rate than did the control group members, although the cash and in-kind treatments are indistinguishable from one another, yielding approximately 6.5% higher completion rates relative to the control group. If the treatment groups had completed cases at the 61.5% rate of the control group, we would have had 175 fewer interviews at the end of Round 10 (about 2% of the NLSY97 sample). The table also shows that the treatment groups exhibited differential behavior both among relatively cooperative respondents who had completed the Round 9 interview and those who had missed the Round 9 interview.

				Table 9-1 Completion Rates in Rounds 10 and 11 Incentive Experiment
	Full Experiment			Control Group			In-Kind Group			Cash Group
	All	Completes		All	Completes		All	Completes		All	Completes
	Cases	Cases	%	Cases	Cases	%	Cases	Cases	%	Cases	Cases	%
Round 10: at end of round
All R10 Experiment Cases	3825	2517	65.8	1293	795	61.5	1266	861	68.0	1266	861	68.0
R9	2390	2037	85.2	784	646	82.4	810	702	86.7	796	689	86.6
not R9	1435	480	33.4	509	149	29.3	456	159	34.9	470	172	36.6
Round 11: at end of round
All R11 Experiment Cases	3825	2579	67.4	1293	805	62.3	1266	877	69.3	1266	896	70.8
R10	2526	2242	88.8	795	686	86.3	865	773	89.4	866	782	90.3
not R10	1299	337	25.9	498	119	23.9	401	104	25.9	400	114	28.5

The lower panel of Table 9-1 shows the preliminary results of the Round 11 incentive experiment. At the end of the round, 2,579 respondents in the experiment had completed an interview. Again, we see that treatment group respondents completed interviews at a higher rate than control group respondents, although cash incentives slightly outperformed in-kind incentives.

The design of this experiment has a strong dynamic component. An advantage of this design is that we can study the effects not only of incentive amounts, but of sequences of incentive amounts on respondent behavior. In the Round 7 experiment, in which we introduced additional payments for missed rounds, we saw the perhaps surprising result that respondents who received increased incentives were more cooperative in subsequent rounds, even though they did not receive any supplemental incentives in those later rounds.

Table 9-2 Incentives Paid [Offered] by Sequence of Completion in Rounds 8-11
			Incentive Paid [Offered if not Paid]
Sequence of completion (C) /non-response (N)		Round		Control/Non-Experimental	In-Kind/ Cash
A. R8C-R9C-R10C-R11C		R9		20	20
		R10		30	50
		R11		30	50
B. R8C-R9N-R10C-R11C		R9		0 [20]	0 [20]
		R10		40	60
		R11		30	50
C. R8C-R9C-R10N-R11C		R9		20	20
		R10		0 [30]	0 [50]
		R11		40	60
D. R8C-R9C-R10C-R11N		R9		20	20
		R10		30	50
		R11		0 [30]	0 [50]
E.R7C-R8N-R9C-R10C-R11C		R9		25	25
		R10		30	50
		R11		30	50
F. R8C-R9N-R10N-R11		R9		0 [20]	0 [20]
		R10		0 [30]	0 [60]
		R11		50	70
G. R7C-R8N-R9N-R10C-R11C		R9		0 [25]	0 [25]
		R10		50	70
		R11		30	50

Table 9-2 describes various patterns of completion and nonresponse across rounds and the sequences of incentives associated with them. For example, we see in the first row that a respondent who has completed each of the Rounds 8 through 11 interviews will have receive $20, $30 and $30 if s/he were a control group or nonexperimental respondent, or $20, $50 and $50 if s/he were in the treatment groups in the Rounds 10-11 experiment.

This table offers some possible insights into the patterns we see in the lower panel of Table 9-1, where Round 9 noninterview respondents returned at a faster rate in Round 10 than Round 10 noninterview respondents returned in Round 11. In rows B and G of table 9-2, we see that respondents who were not interviewed in Round 9 were offered at least twice as much in Round 10 as they had been in Round 9. Among treatment group respondents, the value could be tripled. In contrast, Round 10 noninterview respondents are described in rows C and F. These respondents were generally offered more modest increases of $10 over their most recent (refused) offer. Thus, if we expect the change in incentive amount to affect respondent cooperation, we might expect that Round 10 noninterview returns will be slower than Round 9 noninterview returns.

In the coming months, further analysis of the Round 11 field period will give us additional information about the static and dynamic responses to in-kind and cash incentive supplements relative to our basic respondent incentive. In addition, we will examine data on effort required to work cases and other potential measures of respondent cooperativeness in order to assess more fully the costs and benefits of different incentive levels.

Our preliminary results suggest that an in-kind incentive supplement makes the best use of project dollars to improve respondent cooperation, invest in the unique relationship that each respondent has with the NLSY97, and permit some flexibility in aggregate cost. On that basis, BLS proposes to offer all Round 12 respondents a $30 base fee plus an in-kind incentive of no more than $30 value. We recognize, however, that there are many untapped dimensions to the incentive experiment which may reveal information advocating a different proposal.