Results

Appendix 8
Results of the Panel Maintenance Incentive Experiment

An incentive experiment was conducted during the panel contact information update phase of the ELS:2002 field test in the fall of 2010. The information update phase involved sending letters to sample members and one of their parents requesting that they confirm or update the contact information maintained in the study database. This information included current name, address, and phone numbers and e-mail addresses for the sample member and two parents. A Contact Information Update sheet was included with the letter. It contained the existing contact information from the database to be updated and returned. If the recipients preferred, they could go to the secure study website to confirm or provide updated information. Approximately two weeks after the letter was mailed, a reminder e-mail was sent to each non-responding sample member and their parent.

Half of the field test student sample was randomly assigned to the treatment group (offered a $10 incentive) and the remaining half to the control group (no offer of remuneration). In the panel maintenance reminder e-mail, the sample members in the treatment group were offered a $10 check if they (or their parents) confirmed or updated their contact information. The e-mail to parent alerted them to the incentive offer for their student. Students were mailed the $10 check upon receipt of their confirmation or updated information (regardless of who, the student or the parent, provided the students’ contact information).

The incentive experiment was designed to assess two questions:

1. Would a $10 incentive produce higher overall participation rates during the panel maintenance contact information update than no incentive?

2. Would the $10 incentive produce higher participation rates differentially across subgroups of sample members based on selected characteristics?

Table 1 presents the results of the incentive experiment for the field test sample addressing the first question. Overall, the $10 treatment group had a significantly higher participation rate (25 percent) than the control group (20 percent, one-tailed t = 1.90, p < .05).

Table 1. Panel maintenance participation rates, overall by incentive group.
		Responded
Type of incentive	Eligible sample1	Number	Percent!
$0 (control)	495	99	20.0!
$10 (treatment)	489	124	25.4!

¹ Eligible sample includes sample members for whom an address was known and the mailing was not returned as undeliverable.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2002 Education Longitudinal Study (ELS:2002/12) Third Follow-up Field Test.

Table 2 presents the results of the incentive experiment by selected subgroups addressing the second question. The subgroups examined were selected prospectively based on experience in previous rounds of ELS data collection. They included subgroups who had responded at a lower rate historically as well as subgroups of operational interest such as sample members who participated in previous panel maintenance efforts.

Table 2. Panel maintenance participation rates, characteristics by incentive group.
					Responded
Type of incentive				Eligible sample1	Number	Percent
Second follow-up interview non-respondent
$0 (control)				105	5	3.8
$10 (treatment)				119	10	8.4
Second follow-up interview respondent
$0				390	94	24.1
$10				370	114	30.8*
Second follow-up interview early respondent
$0				197	64	32.5
$10				195	85	43.6*
Second follow-up interview late respondent
$0				127	13	10.2
$10				112	19	17.0
Previous panel maintenance respondent
$0				28	17	60.7
$10				36	27	75.0
Female
$0				238	53	22.3
$10				255	63	24.7
Male
$0				257	46	17.9
$10				234	61	26.1*
Ever indicated high school dropout
$0				32	4	12.5
$10				36	7	19.4
Regular high school diploma (not GED, certificate of completion)
$0				354	87	24.6.
$10				332	109	32.8*.
Postsecondary attendance
$0				313	84	26.8.
$10				303	105	34.7*
Asian/Native Hawaiian
$0				29	6	20.7
$10				28	9	32.1
White
$0				269	73	27.1
$10				246	78	31.7
Black/African American
$0				75	6	8.0
$10				102	15	14.7
Hispanic
$0				97	13	13.4
$10				86	18	20.9

* Difference between response rates of the experimental and control group is statistically significant using a one-tailed t-test.

¹ Eligible sample includes sample members for whom an address was known and the mailing was not returned as undeliverable.

SOURCE: U.S. Department of Education, National Center for Education Statistics, 2002 Education Longitudinal Study (ELS:2002/12) Third Follow-up Field Test.

The following five subgroups demonstrated a significantly higher participation rate (p <.05, one-tailed) for the $10 treatment group than the control group:

• Second follow-up interview respondents (t=2.07);

• Second follow-up interview early respondents (responded in the first 4 weeks) (t =2.28);

• Males (t =2.19);

• Obtained a regular high school diploma (does not include GED, certificate of completion)(t =2.46); and

• Attended a postsecondary institution (t =2.11).

As seen in Table 2, treatment/control differences in participation rates were in the same direction for all subgroups examined, although the differences were not significant for all subgroups.

2. Recommendations

The significantly higher overall participation rate for the treatment group compared to the control group demonstrates the effectiveness of the incentive offer. Every treatment sub-group had a higher participation rate than its control; five of which were significantly different (p < .05). Further evaluation of the overall data from the panel maintenance effort illustrates that the contact information provided by those who responded largely included new information not already in the study database. Specifically, at least one new address, phone number, or email address was provided for either the student or parent (or both) for 82 percent of the cases that responded; indicating that participation was not limited to the easiest-to-locate cases whose information was already current in the database. In addition, 74 percent of cases that responded provided new information for the student as opposed to only providing parent contact information. Being able to make direct contact with the sample student during data collection saves time and costs.

While the incentive did not specifically target historically difficult-to-track cases (e.g., second follow-up interview non-respondents), a significant increase in participation for the treatment group was seen overall and in the five characteristics listed earlier. Increases in participation due to the incentive did not reach significance in all subgroups, but nevertheless occurred in the same direction, with the treatment subgroup demonstrating higher participation rates than the control, for all subgroups evaluated.

Given the increase in participation, the resulting increase in contact data quality, and the relatively low cost compared with using tracing specialists and proprietary databases to locate the sample members, we recommend the implementation of the incentive procedure for use when sample maintenance materials are sent to the full-scale sample in the fall of 2011.

8-1