Appendix A3

The Effectiveness of
Incentives Used in the
Second Phase of a
Two-phase Survey
Charlotte Tubman and Douglas Williams
Presented at the American
Association for Public Opinion
Research Conference
Chicago, Illinois
The views presented in this paper are those of the authors and
do not represent the official views of the United States
Department of Education.

May 14, 2010

National Household Education Survey
(NHES)

• Sponsored by the National Center for
Education Statistics

• Covers learning at all ages
• Surveys roughly every 2 years 1991-2007
• All surveys through 2007: RDD/CATI

2

Motivation for Redesign
• Declining response rates
 Screener response: Low 80’s in early years –
53% in 2007
 Topical rates: 90% in early years – 75% in
2007

• Declining coverage rates for landline
RDD
 January-June 2009: 22.7% of households were
cell-only, and another 1.9% were phoneless
(Blumberg and Luke 2009)
 Exclusion of about 20% of landline telephone
households (Fahimi, Kulp, and Brick, POQ
2009)
3

NHES 2009 Pilot Study

• From RDD to an address based design
• Methodological in nature

• Data Collection Instruments
• Early Childhood Program Participation
(ECPP), for children who have not yet started
kindergarten
• Parent and Family Involvement in Education
(PFI), for children and youth enrolled in K-12

• Encourage participation from
parents/guardians of eligible children
4

Incentive Usage in Surveys
• Rewards offered to encourage
participation or action

• Various Conditions
 Type
• Monetary
• Non-cash

 Timing
• Pre-paid
• Promised

• Used in survey research to address the
issue of survey non-response
5

Review of Incentive Usage in Survey Research
Author

Observed Effects

(Trussell &Lavarkas, 2004; Church,
1993)

-Incentives contribute to significant
increases in response rates

(Shaw et al. 2001; James and
Bolstein, 1992)

-Incentives > $1 significantly increase
response rates

(Nederhof, 1983; James and Bolstein,
1990; Arzheimer & Klein, 1999)

-Correlations between incentives and
respondent characteristics

(Dillman 1991, 2000)

-Incentives paired with multiple
contacts improve mail survey
response rates

6

Incentive Experiment Methodology
• Use of Incentives with the Two-phase Approach
for NHES

 Screener Phase: Non-Experimental
• Households were sent a screener survey that
included a pre-paid $2 incentive
• Returned screeners were used to determine
household eligibility, ECPP or PFI
• Sub-sample of non-response cases were
assigned to receive all future screener contacts
by phone
• If a screener was completed by telephone, the
parent/guardian was asked to complete the
topical survey by phone, if eligible
7

Incentive Experiment Methodology
(cont’d)
• Use of Incentives with the Two-phase Approach for
NHES

 Topical Phase: Experimental
• Mailed topical questionnaires included a
random assignment of either no cash, $5 or
$15
• For topical questionnaires attempted by
phone, a random 50% subsample of those
respondents were offered a promised $5
incentive for their participation

8

Does the inclusion of an incentive have an effect on response rates?

Table 1. Mail Response Rates and Sample Sizes for
Topical Surveys, by Incentive Amount

Total Sampled
Combined
ECPP/PFI
Response Rates

$0

$5

$15

479

510

530

69.5%

73.5%

80.6%*

*Significant ly different from the $0 group at the .05 level

9

Does the inclusion of an incentive have an effect on topical wave response?

Table 2. Conditional Topical Response Rates by initial and
non-response follow-up mailings
$0

$5

$15

Initial Mailing

36.1

40.3

48.4*

1st Follow-up

47.9

46.4

51.2

2nd Follow-up

31.3

30.8

36.5

*Significant ly different from the $0 group at the .05 level

10

Do the characteristics of respondents vary by incentive amount?

11

Do the characteristics of respondents vary by incentive amount?

12

Do the characteristics of respondents vary by incentive amount?

13

Telephone Incentive Experiment
• Subsample of non-response follow-up cases were sent to
the telephone

• Respondents that completed a screener by phone and
had eligible children were randomly assigned into no cash
or $5 promised incentive groups
Table 3. Phone Response Rates and Sample Sizes for
Topical Surveys, by Incentive Amount
$0

$5

Total Sampled for
ECPP/PFI

54

75

Topical Response
Rate

40.7

46.7

14

Overall Findings
•

Incentives are effective in gaining
cooperation at the Topical phase

•

The $15 incentive elicited higher initial
cooperation rates

•

No evidence that offering an incentive
results in respondents with different
characteristics

•

Sample sizes were to small to detect
some potential differences

15

Contact Information
Charlotte Tubman
1550 Research Blvd., TA2065
Rockville, Md 20850
[email protected]

16