Attachment 6 - R18 Incentive Structure Report

OMB Clearance – Round 19 NLSY97

Attachment 6—Report on the Effectiveness of Incentives

R18 Incentive Structure Report

February 7, 2019

1. Introduction

This report summarizes incentives that were implemented in Round 18 of the NLSY97. Each section describes a specific incentive that was paid in the round, and what analyses we have performed to understand the effect of the incentives. If there were any specific experiments conducted during data collection, we specifically describe the results of these experiments in their own subsection.

2. Early Bird Completion

1. Summary

In moving to predominantly telephone data collection, we adopted the Early Bird incentive structure, which has been well-honed on the NLSY79. The premise of the Early Bird premium is to offer respondents the opportunity to call us rather than require us to expend resources to seek them out and gain their cooperation. We split the monetary benefits of the reduced effort by allocating a ‘premium’ to the respondent incentive above the base fee of $40.

The early bird completion rate in Round 18 was roughly 27%. This number is much lower than our experience on the NLSY79, where for instance the early bird completion rate for Round 27 was roughly 38%. Differences in the Early Bird incentive between the NLSY79 and NLSY97 make comparisons difficult. In particular, the NLSY79 has used an Early Bird incentive for many rounds, whereas Round 18 was the first round for which the NLSY97 offered an Early Bird incentive. Therefore, it may be that NLSY79 respondents’ longer experience with the incentive leads to higher take up rates. In addition, the Early Bird amount for Round 27 of the NLSY79 was $30 as opposed to the varying Early Bird amount for Round 18 shown below in Section 2.2. In addition, the base incentive Round 27 of the NLSY79 is higher than that in Round 18 of the NLSY97. This difference may lead to a lack of comparability of Early Bird take-up rates between the two surveys.

Nonetheless, given that there are significant operational fixed costs for system setup and maintenance while fielding as well as costs towards extra staffing needed for CATI shift work and increased high volume respondent communications in order to offer the Early Bird incentive, NORC believes the early bird approach is less valuable with the NLSY97 than it has been with the NLSY79.

2. Early Bird Amount Experiment

Because the responsiveness of the NLSY97 sample to this type of incentive was unknown, and because our introduction of the Early Bird protocol was much less gradual than it was on the NLSY79, we proposed an experiment where we randomly assigned Early Bird offers within each response probability group to determine the relative costs and benefits of different premium amounts.

Key findings from experiment below show that in all cases, the $20 offer resulted in very similar, or sometimes better, completion rates by the end of the round compared to the $35 offer. In addition, cases receiving the $20 offer had similar or better completion rates in the early bird phase. Therefore, because the $20 offer provides similar or better outcomes with lower costs than the $35 offer, we feel that it best balances cost and effectiveness.

Table 1: Early Bird Amount Experiment Results

Response Probability Group	Early Bird (EB) Offer ($)	Total Respondents	Completed Cases	Cases Completed EB	% Complete (Total)	% Complete EB
High	10	1603	1505	513	93.89%	32.00%
High	20	1603	1528	603	95.32%	37.62%
High	35	1603	1504	649	93.82%	40.49%
Medium	10	458	360	81	78.60%	17.69%
Medium	20	917	735	208	80.15%	22.68%
Medium	35	918	736	235	80.17%	25.60%
Low	10	299	74	9	24.75%	3.01%
Low	20	597	152	19	25.46%	3.18%
Low	35	597	138	17	23.12%	2.85%

3. NIR Bonuses

1. Summary

Table 2 below shows the NIR bonus eligibility of various groups of respondents in Rounds 17 and 18, along with their eventual completion rates. Individuals are assigned to different NIR bonus amounts in either round based on their interview history; this analysis provides only descriptive evidence on the relative efficacy of NIR bonuses.

In terms of the relative pattern of responses in Rounds 17 and 18, we see the individuals out 3 or more rounds respond at higher rates in Round 18. While this may suggest that the additional NIR bonuses in Round 18 served to increase response rates in this group, it is difficult to compare these groups directly between Round 17 and Round 18 because the composition of the groups changes between rounds. For instance, some of the respondents out 4 rounds in Round 17 have now entered the Round 18 “out 5 or more rounds” group. Nonetheless, the overall patterns of responses across these groups are similar between rounds. For both rounds, respondents who completed last round have very high response rates near or above 90 percent in both rounds. Completion rates drop substantially for respondents out only one round, and continue to drop to the point where the completion rate for respondents out 5 or more rounds is near 7 percent in both Round 17 and Round 18. Therefore, the NIR bonuses may have helped us recruit respondents in Round 18 and Round 17.

Table 2 also shows Early Bird and in person completion rates for these different groups. While Round 17 did not offer an Early Bird incentive, the results from Round 18 show that the majority of Early Bird completes occurred for individuals who completed last round. Early Bird completion rates for individuals who had been out at least one round were very low.

The results for fraction of completes that were made in person are not comparable between Round 17 and Round 18 because the survey moved from in-person to phone as the primary mode of collection in Round 18. Still, the patterns of relative in person completion rates across group of respondents who have been out rounds is similar in both rounds. This again suggests that the bonuses functioned similarly in Round 17 and Round 18.

Table 2: NIR Bonus Summary Table

Status	R17 NIR Bonus	R18 NIR Bonus	R17 N	R18 N	R17 Completion Rate	R18 Completion Rate	R18 EB Completion Rate	R17 Fraction of Completes In Person	R18 Fraction of Completes In Person
Completed Last Round	0	0	7141	7103	93.63%	89.65%	32.23%	74.99%	10.17%
Out 1 Round	15	20	596	455	45.47%	37.14%	4.40%	48.71%	16.56%
Out 2 Rounds	30	40	183	325	32.79%	24.31%	3.38%	36.66%	17.73%
Out 3 Rounds	45	60	155	123	16.13%	26.83%	4.88%	44.02%	12.11%
Out 4 Rounds	45	60	95	130	6.32%	17.69%	3.08%	16.61%	4.35%
Out 5 Rounds*	45	-	814	848	6.76%	7.31%	0.47%	38.17%	19.43%
*In Round 18, respondents out 5 or more rounds were eligible for the “Big NIR” bonus (see below).

2. Big NIR Bonus Experiment

As an experiment, participants who had not responded in 5 or more rounds were offered an additional large bonus if they responded in Round 18. The table below summarizes the results of the experiment. Note that those who had responded in the last 5 rounds are categorized as ineligible. This includes all participants in the high and medium response probability groups as well as a portion of the low probability groups.

Within the portion of the low probability group who had not completed in five or more rounds, the remaining participants were split into two categories: a control group who were potentially still eligible to receive the other bonuses outside of the Big NIR Bonus, and a treatment group who were eligible to receive the Big NIR Bonus as part of the experiment. Within the low response probability group, those who received the NIR bonus completed the survey at a higher rate than those in the control group (15.88% vs 8.48%). Therefore, we feel this bonus was successful at raising response rates for this group of respondents who we traditionally have great difficulty getting to respond.

In future rounds, we plan to evaluate whether this Big NIR bonus leads to long term benefits. In particular, it will be especially useful to use future completion data to evaluate whether respondents who returned to the survey due to the Big NIR bonus are more likely to complete in future rounds.

Table 3: Big NIR Bonus Results

NIR Bonus Received	Total Respondents	Completed Cases	Cases Completed EB	Cases Completed in Person	% Complete (Total)	% Complete EB	% Complete In Person
Control	389	33	4	6	8.48%	1.03%	1.54%
Treatment	170	27	1	6	15.88%	0.59%	3.53%

3. Talk to Us

The “Talk to Us” bonus offers went out over the course of 3/26/2018 – 3/28/2018 to a little under 400 cases. The selected individuals received an email, text or call inviting them to respond to NORC for a $5 Walmart card to be provided electronically. The hope was that this additional incentive would allow interviewers to encourage individuals to make contact with NORC and eventually complete the interview.

Table 4 below shows the results of this bonus. “Contacted us” refers to taking up the bonus, where responses were roughly split half and half between texts or emails. These results show that while the bonus was successful in encouraging respondents to contact us, it did not result in measurably larger completion rates. Therefore, while this option allows NORC another “arrow in the quiver” when trying to engage respondents and may help us to gain updated contact information, the increase in completes is negligible and we are not pursuing this option for Round 19.

Table 4: Talk to Us Bonus Results

	% Contacted Us	% Completed During Round	N
Bonus offered	23.9%	32.2%	364
Bonus not offered	1.4%	30.2%	376

3. Final Push

Starting after the first 12 weeks of the round, cases that had at least 7 contact attempts or at least one refusal were eligible for a final push incentive of $20 in both Round 17 and Round 18 .  In order to facilitate a timely close to the fielding, starting 6 months after the start of fielding, all cases were eligible for this incentive.  In both Round 17 and Round 18, almost all final push incentives were received after the first six months of the field period.

Evaluating the final push incentive is difficult given that it was not designed to be evaluated in a rigorous manner. In particular, given that the final push is essentially a bonus that applies to all respondents in the survey after a given time in the data collection, there is no suitable comparison group with which to perform a rigorous evaluation.

To provide some information on the effects of the final push bonus, we examine response rates in the months before and after the final push goes into effect. Table 5 shows these results for both Round 17 and Round 18, where the percentages reflect the number of completed cases during a month divided by the number of cases that had not responded at the start of the month. If these numbers show increases over the month, it would suggest the final push increased response rates. These results show a decrease in the conditional response rate after final push in Round 17, but in Round 18 there was an increase in this response rate of roughly 1 percentage point. Again, it is difficult to do a proper evaluation of this incentive given the lack of a suitable comparison group. Nonetheless, this is suggestive evidence that the final push did contribute to the final stages of production (either to slow decreases or actually increase production), and likely was a greater contributor to Round 18 completions than the same incentive design was for the prior round.

Table 5: Response Rates Relative to Final Push

	Fraction of Remaining Cases Completed
	Month Before Final Push	Month After Final Push
Round 17	10.42%	8.05%
Round 18	16.89%	17.70%

Table 6 provides information on the number of contacts relative to the final push date for three groups of respondents: those who completed the month after the final push took effect, those that completed more than a month after the final push took effect, and those than never completed. For respondents who complete the month after the final push takes effect, we see a notable increase in the number of contacts in Round 17 from 1,215 to 1,757, but the number of contacts in Round 18 remains relatively flat at 5,617 versus 5,770. For respondents that eventually complete more than a month after final push goes into effect and respondents that never complete (regardless of completion date), we see similar or fewer contacts in the month after final push compared to the month before final push. While these are again descriptive facts and cannot be used to evaluate the final push, they imply that at least for Round 18, completion rates raised slightly after final push took effect without a pronounced uptick in the number of contacts. Given that case completion gets progressively harder as data collection progresses, these results suggest that the final push contributed to the final stages of production.

Table 6: Contacts by Month Relative to Final Push

			Total Contacts
Round	Group	N	Month Before Final Push	Month After Final Push
Round 17	Complete in month before final push	289	2742	N/A
	Complete in month after final push	200	1215	1757
	Do not complete in month after final push, but complete eventually	403	2061	2127
	Never complete	1881	6615	5194
Round 18	Complete in month before final push	806	7094	N/A
	Complete in month after final push	702	5617	5770
	Do not complete in month after final push, but complete eventually	1013	6745	5906
	Never complete	2250	9096	6580

3. Electronic Payments

In Round 17, we introduced the electronic service PayPal as a payment option for respondents who completed the interview by phone. PayPal allowed the project to reduce administration fees, improve the speed of paying the respondent, confirm contact information (such as mobile phone number or e-mail address to which payment can be made), and allow the project to be proactive about resolving issues related to the transaction. Table 7 shows the amount of respondents selecting electronic payments by round. The Round 17 experience with PayPal was a 17% overall rate of selection of PayPal by telephone respondents as their choice of payment method when completed by phone. With the use of a $5 incentive in Round 18 and expansion to include Chase QuickPay as well as PayPal, the number went up to roughly 40%. We feel that these results show promise in increasing the use of electronic payments.

Table 7: Electronic Payments Bonus

Electronic Payment	Percent of Phone Interviews Receiving Payment Electronically
Round 17	17%
Round 18	40%