Appendix A2

Screening for Specific
Population Groups in Mail
Surveys
Douglas Williams, Jill M. Montaquila,
J. Michael Brick, Mary C. Hagedorn
Presented at the 65th Annual AAPOR Conference
Chicago, Illinois

The views presented in this paper are those of the authors and do
not represent the official views of the United States Department of
Education.

May 15, 2010

Introduction

Screening for Specific Population Groups
• In cross-sectional surveys generally conducted
through RDD
 Inexpensive method of screening large number of
households
 May need to sample from within households
 Attractiveness of completing the topical survey during the
screening contact

2

Introduction (Cont.)

• Problems with RDD screening approaches
 Declining response rates: (Steeh et al. 2001; Curtin,
Presser and Singer 2005; Battaglia et al. 2008)
 Declining coverage rates for landline RDD
• January-June 2009: 22.7% of households were cellonly, and another 1.9% were phoneless (Blumberg
and Luke 2009)
• Exclusion of about 20% of landline telephone
households (Fahimi, Kulp, and Brick, POQ 2009)

3

Introduction (Cont.)

• Using mail as an alternative: single-phase
 Use Address Based Sample – increasing population
coverage relative to landline RDD (Fahimi; Presented at
2010 FedCASIC workshop)
 Much of the questionnaire is irrelevant to most sampled
addresses
 Uncontrolled sampling

4

Introduction (Cont.)

Two-phase Mail Approach
• Mail screener to all households
• Use returned screeners to determine eligibility and
perform sampling
• Mail topical survey only to sampled persons (in
eligible households)
• Nonresponse bias is a concern
 Education/Income biases with mail mode
 Potential for differential response between target and nontarget population
5

Introduction (Cont.)

The National Household Education Surveys Program
• Sponsored by the National Center for Education
Statistics

• Targets households with children
• Surveys roughly every 2 years 1991-2007
• All surveys through 2007: RDD/CATI
• Declining response rates
 Screener response: Low 80’s in early years – 53% in 2007
 Topical rates: 90% in early years – 75% in 2007

• 2009 Pilot Study of address-based sample
6

Overview of Design

• NHES targets households with children
• Key elements of screener experiment – screener
versions tested
 Screen-out: 1 page 11x17; only asked about children
 Core: 4 pages 8½x11; asked about children and 9
household questions
 Engaging: 6 pages 8½x11; same as core with
additional 16 questions on education

7

Overview of Design (Cont.)

• Purpose of screener versions
 Does a child-focused survey request (screen-out) result in
different response propensities for households with
children?
 Does increasing content of the survey (engaging) to
engage the respondent result in different response
propensities for all households?
 Does screener version result in different response
propensities to the topical survey?
 Does screener version affect the composition of
respondents?

8

Overview of Design (Cont.)

• Key elements of screener experiment
 Two-phase data collection
• Screener
• Topical survey (personalized)

 Test in a national sample (n = 10,200)
 Mail contacts based on general procedures outlined by
Dillman et al. (2009)

9

Key Findings

•

Response rates: National Sample

National sample rates
Overall Screener response rate

Screenout

Core

Engaging

63.6%*

58.3%

60.1%

* Significantly different from core and engaging versions p < 0.05

10

Key Findings (Cont.)

•

Response rates: National Sample

National sample rates

Screenout

Core

Engaging

Overall Screener response rate

63.6%*

58.3%

60.1%

Initial Mailing only

28.3%*

23.2%

23.2%

1st follow-up mailing

25.8%

23.6%

23.6%

2nd follow-up mailing (FedEx)

31.8%

29.4%

32.5%

* Significantly different from core and engaging versions p < 0.05

11

Key Findings (Cont.)

•

Proportion of households identified with eligible
children: National Sample


Estimates from ACS suggest this should be ~35%

National sample rates
Overall Proportion w/children

Screenout

Core

Engaging

30.6

31.5

32.7

12

Key Findings (Cont.)

•

Proportion of households identified with eligible
children: National Sample

National sample rates

Screenout

Core

Engaging

Overall Proportion w/children

30.6

31.5

32.7

Initial Mailing only

25.8

26.2

31.4

1st follow-up mailing

33.1*

34.8*

31.2

2nd follow-up mailing (FedEx)

37.0*

35.2*

35.6

*significantly different from initial mailing of same version p < 0.05

13

Key Findings (Cont.)

•

Response rates: Targeted Sample (n = 800) of
households identified by sample vendor to
included children.

Targeted sample rates
Overall Screener response rate

Screenout

Core

Engaging

80.7%*

70.3%

71.9%

* Significantly different from core version p < 0.05

14

Key Findings (Cont.)

•

Proportion of households identified with eligible
children: Targeted Sample

Targeted sample rates
Overall Proportion w/children

Screenout

Core

Engaging

80.7%

83.5%

82.5%

15

Key Findings (Cont.)

•

Response to topical survey (2nd phase response)

National sample

Screenout

Core

Engaging

Topical Response

73.1

74.8

76.3

Targeted sample

Screenout

Core

Engaging

Topical Response

86.4

85.2

85.3

16

Key Findings (Cont.)

•

Sensitive Questions – Phone number & Child’s
First Name

National sample
Percent providing phone
number

Screenout

Core

Engaging

N/A

69.9*

56.7

* Significantly different from engaging version p < 0.05
National sample

Screenout

Core

Engaging

First Name

77.5*

75.1*

61.5

Initials/Nickname

20.4

20.7

30.8

No Name

2.1

4.2

7.7

* Screen-out and core significantly different from engaging version p < 0.05

17

Key Findings Topical Respondents

•

Homeownership

18

Key Findings Topical Respondents (Cont.)

•

Education

19

Key Findings Topical Respondents (Cont.)

•

Income

20

Key Findings Topical Respondents (Cont.)

•

Presence of non-English Speaking household
members

21

Summary

• Overall, screening for households with children by
mail in a two-phase approach was successful
• Pilot was too small to detect some potential
differences
• 2011 larger methodological Field Test
 Screenout & Engaging version
 Test request for child’s name vs. no name

22

Contact Information

Douglas Williams
1600 Research Blvd., RE 457
Rockville, MD 20850
[email protected]

23

24

Key Findings Topical Respondents (Cont.)

•

Cell only and no-phone households

25

Key Findings

• Pilot study response compared to NHES:2007
 Screener response rate: 59% vs. 53% in NHES:2007
 Topical response rate: 74% vs. 75% in NHES:2007
 Process worked, people gave information on their children

26