Appendix A1

Maximizing Response in a
Two-Phase Survey with
Mail as the Primary Mode
Jill M. Montaquila, J. Michael Brick,
Mary C. Hagedorn, Douglas Williams

Presented at the 65th Annual AAPOR
Conference
Chicago, Illinois

The views presented in this paper are those of the authors and
do not represent the official views of the United States
Department of Education.

May 14, 2010

Outline

• Introduction
 The National Household Education Surveys Program
(NHES): Background
 Motivation for redesign

• Overview of design
• Take-home message

• Embedded experiments
• Key findings

• Plans for the 2011 Field Test
• Summary
2

Introduction

The National Household Education Surveys Program
• Sponsored by the National Center for Education
Statistics

• Surveys roughly every 2 years 1991-2007
• All surveys through 2007: RDD/CATI

• Within-household sampling (up to 3 persons)
• Sample sizes:
 34,000 – 64,000 completed household Screeners
 2,250 – 24,600 completed Topical surveys

3

Introduction (Cont.)

Motivation for redesign
• Declining response rates
 Screener response: Low 80’s in early years – 53% in 2007
 Topical rates: 90% in early years – 75% in 2007

• Declining coverage rates for landline RDD
 January-June 2009: 22.7% of households were cell-only,
and another 1.9% were phoneless (Blumberg and Luke
2009)
 Exclusion of about 20% of landline telephone households
(Fahimi, Kulp, and Brick, POQ 2009)

4

Overview of Design

• 2009 Pilot Study to be followed by large-scale
methodological Field Test in 2011
• Pilot Study objectives
 Alternative to landline RDD
 Must be feasible for a full-scale NHES collection
 Rule out approaches that clearly were not worthwhile-NOT to detect small differences
 Work out operational kinks
 Determine any needed schedule adjustments

5

Overview of Design (Cont.)

• Key elements of Pilot Study design
 Address-based sample (n=10,200)
 Mail as primary mode, with test of phone follow-up
 Two-phase collection
• Screener
• Topical survey (personalized)

 Embedded experiments
 Three samples
• National sample (n=10,200)
• Sample of addresses in linguistically isolated (Spanish) tracts
(n = 800)
• Targeted sample of households with children (n = 800)

6

Take-home Message

• Overall, Pilot Study provided evidence that the twophase self-administered approach is feasible
 Screener response rate: 59% vs. 53% in NHES:2007
 Topical response rate: 75% (same as NHES:2007)
 Certain conditions yield even higher rates
• As high as
– 64% for Screener (all-mail cases; screenout),
– 85% for Topical ($15, FedEx follow-up, all-mail Screeners)



Based on preliminary examination of key items
•
•
•

Item response rates were high
Little evidence of skip pattern issues
Passed “laugh test” comparison for key statistics

7

Embedded Experiments: Screener Phase

Engaging
Screener - $2

Core
Screener - $2

Screenout
Screener - $2

Thank you
postcard

1st follow-up
Same Scnr as
1st mailing

FedEx
original scnr

1st follow-up
Phone*
Scnr + Topical

Phone*
Scnr + Topical

Reduced
(nonresponse)
Screener

8

Embedded Experiments: Topical Phase

Completed mail
screener

1st Topical
Mailing

$0

$5

Completed phone
screener

$15

Phone
Topical not
completed at scnr
contact

Thank you
postcard

Priority Mail

FedEx

Phone

9

Key Findings

•

Screenout and Engaging Screeners
outperformed the Core Screener

National sample rates

Screenout

Core

Engaging

Screener response rate

61.8%

56.9%

57.2%

% of households with children

30.1

30.5

32.2

Topical response rate

73.1

74.8

76.3

10

Key Findings (Cont.)
• Screener follow-up stages picked up households

with children at higher rates than the initial stage
Respondent subgroup

Percent of households with
children

Overall

30.9

Initial respondents

27.4

Follow-up respondents
Mail only for 1st and 2nd follow-up

34.8

Phone for 1st or 2nd follow-up

30.5

•(last figure should be 30.4)
More in Williams et al. (AAPOR 2010, Saturday
2:15 p.m., Concurrent Session C)
11

Key Findings (Cont.)

•

Mail outperformed phone for follow-up


For Screener nonresponse, limited by ability to match
phone numbers (57% match rate)
Even when phone number matches were available,
phone follow-up was much less effective than mail


•
•

19% of Screener cases assigned to phone follow-up
finalized as nonworking/nonresidential
Only 17% of mail Topical nonresponse cases sent to phone
for follow-up were completed

12

Key Findings (Cont.)

•

Mail outperformed phone for nonresponse followup (cont.)

Assigned mode for Screener
follow-up

1st follow-up
(Screener)
completion rate

2nd follow-up
(Screener)
completion rate

Mail (No vendor phone number)

42.8

27.1

Mail (Vendor phone number)

49.3

34.6

Phone (Vendor phone number)*

34.4

21.8

Phone (Vendor phone number),
completed by phone

18.0

12.4

*Includes late mail returns received after case had been sent for telephone collection

13

Key Findings (Cont.)

• Incentives were effective in eliciting cooperation at
the Topical phase
 Topical response rates by incentive level:
• $0: 70%
• $5: 74%
• $15: 81%

 No evidence that offering an incentive results in respondents
with different characteristics, but the $15 incentive elicited
higher initial cooperation rates
 More on this in Tubman and Williams (AAPOR 2010, Today,
1:45 p.m., Concurrent Session C)

14

Key Findings (Cont.)

• For Topical nonresponse follow-up, indication that
FedEx might outperform Priority Mail
• 53% of those followed up by FedEx responded,
compared to 49% for Priority Mail (not significant
at α=0.05)

15

Key Findings (Cont.)

Linguistically Isolated Sample
•

Experimented with English-only (Core) and
Bilingual Screeners

•

Similar response rates for English and Bilingual
Screeners:
46.2% (English) vs. 45.8% (Bilingual)

•

Language appears to be a key real barrier to
participation (More on this in Zukerberg and
Han, AAPOR 2010, Thursday, Poster Session 1)

16

Plans for the 2011 Field Test

• Mail as primary mode for Screener and Topicals;
very limited use of telephone
• Screener form/material experiments
 Asking for child’s name vs. not asking
 English only vs. bilingual vs. dual (English and Spanish) to
Spanish surname/linguistically isolated

• Mailing service experiments
• Topical incentive experiments

17

Summary

• For NHES, transition to address-based sampling
with mail as primary mode is promising
 Higher response rates than last RDD collection
 Much higher coverage rates than landline RDD
 No apparent significant data quality issues

• Challenges (particularly language and literacy)
remain
• Methodological experiments in 2011 Field Test will
inform design of the future

18

Contact Information

Jill M. Montaquila
1600 Research Blvd., RE 482
Rockville, MD 20850
[email protected]

19