JSM2008 Proceedings Bias Study Research Paper

An Investigation of the Potential Effects of Nonresponse
on the BTS Omnibus Household Survey
Pheny Z. Weidman
Bureau of Transportation Statistics/Research and Innovative Technology Administration, U.S. Department of Transportation, 1200 New Jersey Avenue, SE, Washington, DC 20590

Abstract
The Bureau of Transportation Statistics’ Omnibus Household Survey (OHS) is a Customer Satisfaction Survey using a
list-assisted random digit dialing (RDD) methodology. It assesses the general public's perception of, expectations from,
and satisfaction with the nation's transportation system by interviewing persons in randomly selected telephone
households. The response rate for the OHS has been around 50% since its inception in 2000, raising concerns about the
generalizability of its results.
This study examined if and how nonresponse introduces bias into the OHS estimates. It used a modified form
of nonresponse follow-up approach that compared response rates of early versus late respondents to investigate the
potential effects of nonresponse on major OHS estimates. It was relatively cheap. The results and methods used in this
study can provide useful information to other government agencies regarding the conduct of nonresponse bias studies
to meet the Office of Management and Budget’s requirements for their own RDD surveys.

Key Words: Survey nonresponse, nonresponse bias, RDD survey

1. Introduction
Survey nonresponse bias is defined as the departure of the expected value of an estimate from its true value due to
nonresponse (Groves, 2006). For any given survey, two types of nonresponse may occur--unit and item nonresponse.
Unit nonresponse occurs when a sampled unit such as a person, household, or organization fails to respond to a survey.
Item nonresponse refers to the situation in which a unit response is obtained, but the respondent omits answers to some
questions on the survey (Dillman et al., 2002).
It has been well-documented that response rates to social and behavioral surveys have been declining in recent decades
(Johnson et al., 2006; Abraham et al., 2006; de Leeuw and de Heer, 2002; Curtin et al., 2005; Atrostic et al., 2001).
This phenomenon has caused growing concern among survey researchers regarding the quality of survey data.
Although government survey response rates generally tend to be higher than those for private surveys, U. S. federal
statistical agencies nonetheless have experienced increasing difficulty in obtaining household survey interviews
(Abraham et al., 2006). The Bureau of Transportation Statistics (BTS)’ Omnibus Household Survey (OHS) is a case in
point. Despite the survey’s official imprimatur and an extensive follow-up effort, the OHS response rate has been
around 50 percent. It has naturally raised concerns within the BTS regarding whether and how the data obtained can be
generalized to the target population.
The OHS employed the list-assisted random digit dialing (RDD) method to draw a national probability sample of
households from the United States non-institutionalized adult population (18 years of age or older). For this mode of
data collection, there is no information available regarding nonresponse households other than their telephone numbers
and a call history maintained by the interviewers. This makes the study of effects of nonresponse extremely
challenging.
A review of the literature on nonresponse in household survey research revealed several methodologies commonly
employed to investigate the potential effects of nonresponse on survey estimates from RDD surveys. One of the
common methods for analyzing nonresponse bias is to compare the respondent-based estimate with those from another
source considered to be more accurate (Groves, 2006). In the case of household surveys, this approach compares the
distributions of age, gender, race, and other sociodemographic variables among respondents with those from the most

recent census of the population. The limitations of this approach are that the key survey variables of the study often do
not exist in the other source; that different forms of measurement may be used by the focal survey and the benchmark
data source or survey; and that the coverage and nonresponse characteristics of the benchmark data source or survey are
not completely known. In the case of the 2006 OHS, the most recent decennial census data available were collected
more than six years ago and would be too out-of-date to use as variables predictive of survey nonresponse.
Another class of nonresponse bias studies is the nonresponse follow-up study. It is a two-phase survey approach,
which tries to conduct follow-up interviews with persons not responding to the initial survey (Johnson et al., 2006;
Lahaut et al., 2002; Gmel, 2000; Hill et al., 1997). Some studies investigate variation within the existing survey. This
class of techniques actually consists of modified forms of the nonresponse follow-up study. Examples of this include
comparing estimates of early respondents to those of late respondents (Johnson et al., 2006; Voigt et al., 2003; Etter
and Perneger, 1997; Triplett et al., 1996; Kristal et al., 1993; Lavrakas et al., 1992), comparing estimates from early
cooperators with those from the full respondent data set (Groves, 2006; Curtin et al., 2000, 2005; Lin and Schaeffer,
1995; Dunkelberg and Day, 1973), comparing respondents in the first phase sample with those in both the first and
second phase samples (Groves and Wissoker, 1999), and comparing observations made during data collection on both
respondents and non-respondents (Groves and Couper, 1998). These approaches are limited by the question of how
representative responses from the follow-up or reluctant respondents are of all nonrespondents, and by the frequent use
of multiple or different data collection methods during the follow-up phase of study that may confound nonresponse
error with measurement quality (Johnson et al., 2006). In addition, there usually is additional, some time even very
expensive, cost associated with conducting the follow-up phase of study.
Since Pace used it in 1939, comparisons of early versus late respondents to survey requests had been accepted and used
by statisticians and survey researchers as an analytical method of studying survey nonresponse bias. In this approach,
researchers usually classify survey respondents into either early or late responders by using indicators of difficulty of
eliciting a response. Early studies that employed this method were almost exclusively based on data from postal
surveys (Lynn et al., 2002; Dunkelberg and Day, 1973; Filion, 1976; Hawkins, 1975; Ellis et al., 1970; Larson and
Catton, 1959; Hilgard and Payne, 1944; Reid, 1942; Shuttleworth, 1940; Suchman and McCandless, 1940; Pace, 1939).
The number of mailings or elapsed days prior to response was used as the indicator in these studies. In more recent
studies based on telephone interview surveys, researchers typically used the total number of interviewer call attempts as
the indicator of difficulty (Lynn et al., 2002; Cheng, 1998; Traugott, 1987; Thomsen and Siring, 1983; Fitzgerald and
Fuller, 1982; Drew and Fuller, 1980). Because of the limitations of the OHS design mentioned in the first section of
this paper, the method of nonresponse bias study that is appropriate for OHS is restricted to this class of approaches.

2. Data and Method
The OHS is conducted by the BTS with joint funding from the Transportation Security Administration (TSA). It
assesses the general public's perception of, expectations from, and satisfaction with the nation's transportation system.
Data were collected from households in the U.S. using a Random-Digit-Dialed (RDD) telephone methodology. The
survey was first conducted in 2000. The analysis this study reports is based on OHS data collected during Nov. 2006.

2.1 Sample Design
The target population for the OHS is the noninstitutionalized population aged 18 years or older who are currently living
in the United States. To ensure that a sample of telephone numbers is geographically representative, telephone prefixes
are stratified by their associated Census Bureau divisions and metropolitan status (Bureau of Transportation Statistics,
2006). The rates at which households are sampled for the OHS differ across these strata. In the last stage of sample
selection, one randomly selected person age 18 or older in each sampled household is designated for participation in the
OHS. The list-assisted random digit dialing methodology was employed to generate the desired sample. List-assisted
refers to the use of commercial lists of directory-listed telephone numbers to increase the likelihood of dialing
household residences. This method gives unlisted telephone numbers the same chance to be selected as directory-listed
numbers.
The 2006 OHS interviews were conducted between the first and the last day in November, 2006. If a selected person
could not be contacted or was not available, an interviewer would try to call back later at a different time. Although the
maximum number of calls back to a non-contact person was set as 60, the highest number recorded in our data was 80.
The disposition of each call was recorded by the interviewers.

The OHS final analysis weight is the product of five components: 1) Base sampling weight; 2) Adjustment for unit
nonresponse; 3) Adjustment for households with multiple telephone numbers; 4) Adjustment for selecting an adult
within a sampled household; and 5) Post-stratification adjustments to the target population.

2.2 Methods
To examine the potential effects of nonresponse, first 2006 OHS interview history and disposition data were merged
with the OHS data file. Indicators of the level of difficulty of eliciting a response based on each sample’s calling
history were then constructed for the 1,095 sample units in the 2006 OHS. Initially, each OHS sample was classified
into one of three categories: respondent, difficult to contact, and refusal. Because the OHS total sample size was so
small, in order to ensure enough sample size for subsequent analyses of nonresponse bias of key OHS estimates, only
two categories -- respondent and late respondent/refusal -- were used in final analyses. Individuals who completed
telephone interviews by no later than the 4th phone call were classified as respondents. Those who completed telephone
interviews only after either initially refusing to participate or repeated (at least five) phone calls were classified as late
respondents/refusals. Individuals with missing call history information were randomly assigned into one of the two
categories.
When survey nonresponse bias is studied by investigating variation within the existing survey, such as comparing
estimates of early respondents to those interviewed only with great effort, an important assumption is made or implied.
It is assumed that those reluctant respondents are representative of non-respondents (Johnson et al., 2006; Voigt et al.,
2003; Etter and Perneger, 1997; Triplett et al., 1996; Kristal et al., 1993; Lavrakas et al., 1992). Based on this
assumption, we created a second set of weighted OHS data. The unit nonresponse adjustment component in the new
OHS’ final weight reflected the assumption implied by the analytical method. That is, those late respondents/refusals
were treated as nonrespondents when adjusting for unit nonresponse. When assumptions about data were true, for each
OHS measure, the differences between OHS estimates derived from weighted samples using original and new final
analysis weights were due to survey nonresponse bias.
Nonresponse bias was estimated in five core OHS measurement areas: A) mode of transportation used for journey to
work, B) method used to improve commute, C) Telework, D) opinion on traffic congestion, and E) opinion on traffic in
general. The specific core OHS measures included in this study were:
(1) percentage of population worked outside home;
(2) percentage of population used non-company vehicle alone for journey to work;
(3) percentage of population used non-company vehicle with others for journey to work;
(4) percentage of population used carpool or vanpool for journey to work;
(5) percentage of population used buses for journey to work;
(6) percentage of population walked to work;
(7) percentage of population changed schedule/work hours to improve commute;
(8) percentage of population moved closer to work to improve commute;
(9) percentage of population worked at home to improve commute;
(10) percentage of population used a toll road/lane to improve commute;
(11) percentage of population whose work could be done at home;
(12) percentage of population for whom Telework was an option;
(13) percentage of population worked at home last week;
(14) people’s opinions on traffic in general; and
(15) people’s opinions on traffic congestion.
The Taylor series linearization method for a Stratified Without Replacement (STRWR) sample design was used for
variance estimation in this study.

3. Findings and limitations
3.1 Findings
The analyses results in table 1 present the relative nonresponse biases for OHS estimates of the thirteen dichotomous
OHS core measures (1) through (13). The relative nonresponse biases of these OHS estimates ranged from zero when
estimating the “percentage of population used non-company vehicle with others for journey to work” to 4.82% when

estimating the “percentage of population for whom Telework was an option.” The signs in the relative bias column
indicate the directions of the biases. A positive relative bias indicates an overestimate of the target population value
whereas a negative relative bias indicates an underestimate. For example, a 4.82% relative bias for the measure of
percentage of population for whom Telework was an option meant that the estimate of this measure based on
unadjusted respondent data (original OHS sample) had overestimated the target population value by about 4.82%.

Table 1: Nonresponse Bias in the 2006 BTS Omnibus Household Survey
Selected
core
transportation
measurements/questions

New
Sample
Estimate

Standard
Error

Sample
Estimate

Standard
Error

Relative
Bias (%)

(1) Worked outside home

59.24

2.24

57.85

1.81

-1.39

Mode of transportation used for
journey to work
(2) Non-company vehicle alone
(3) Non-company vehicle with
others
(4) Carpool or vanpool
(5) Bus
(6) Walk

78.15

2.30

78.43

2.07

0.28

13.22
5.27
5.77
9.16

1.84
1.27
1.38
1.72

13.22
4.54
5.54
9.32

1.65
1.03
1.20
1.53

0.00
-0.73
-0.23
0.16

Method used to improve commute
(7) Changed schedule/work hours
(8) Moved closer to work
(9) Worked at home
(10) Used a toll road/lane

17.82
10.44
9.98
5.40

2.02
1.87
1.62
0.97

18.80
9.41
10.63
5.88

1.83
1.41
1.45
0.94

0.98
-1.03
0.65
0.48

Telework
(11) Work could be done at home
(12) Telework was an option
(13) Worked at home

25.29
60.10
32.44

2.68
6.48
6.60

23.75
64.92
32.02

2.17
5.16
5.63

-1.54
4.82
-0.42

Table 2: Opinions on Traffic and Congestion
Percent of Reporting

Opinions on

New
Sample
Estimate

Standard
Error

Sample
Estimate

Standard
Error

Relative
Bias (%)

(14) Traffic
Much better
Better
Unchanged
Worse
Much worse

2.78
6.81
62.59
16.86
10.96

0.7
1.46
2.5
1.84
1.56

2.97
7.37
62.41
17.06
10.19

0.71
1.38
2.26
1.70
1.35

0.19
0.56
-0.18
0.20
-0.77

(15) Congestion
Very congested

17.19

2.11

16.62

1.82

-0.57

Moderately congested
Slightly congested
Not at all congested

30.86
21.49
30.46

2.54
2.1
2.29

30.06
21.80
31.51

2.17
1.92
2.12

-0.80
0.31
1.05

Table 2 presents the results of relative nonresponse biases analyses in estimates of two core OHS measures that are
polychotomous, (14) and (15). These two OHS measurements were designed to measure people’s opinions on traffic
in general as well as on traffic congestion specifically. The results in Table 2 indicate that the relative nonresponse bias
in the estimate of each measurement category was very small, ranging from 0.18 to 1.05.
Overall, our analyses show that nonresponse biases of all key OHS estimates are very small. In most cases, the size of
the nonresponse bias was around 1%. The measurement of “whether Telework is an option” had the largest
nonresponse bias, 4.8%, when analyzed with our method.

3.2 Limitations of the Study
There are several limitations in this study. They can be classified as being associated with either the analytical method
or data used by the study.

3.2.1 Method Limitations
Limitations of nonrespondense follow-up studies have been briefly discussed previously in this article. The main
weakness of the approach used is that non-respondents to the survey are excluded from the study. This approach is also
limited by the question of how representative reluctant respondents are of all non-respondents (Johnson et al., 2006).

3.2.2 Data Limitations
There are two major limitations associated with the 2006 OHS data used in this study. It is a very small sample with
only 1,095 cases. Whether this is large enough to make any estimates of the U.S. population is questionable. In
addition, for this particular study, the missing data rate was very high. About 17.5% of the cases didn’t have call
history information.
Despite its limitations, this study provided nonresponse bias of key OHS estimates for the first time since the survey’s
inception in 2000. It was relatively cheap. Although findings of the study were in the survey’s favor, it is
recommended that more nonresponse bias studies are needed. The author suggests that future studies of nonresponse
bias in the OHS use multiple methods simultaneously whenever possible and /or multiple years of data.

References
Abraham, K. G., Maitland, A., and Bianchi, S. (2006), Nonresponse in the American Time Use Survey: Who is missing
from the Data and How Much Does It Matter? Public Opinion Quarterly, Vol. 70, No. 5, Special Issue, pp. 676–
703
Atrostic, B. K., Bates, N., Burt, G, and Silberstein, A. (2001), “Nonresponse in U.S. Government Household Surveys:
Consistent Measures, Recent Trends, and New Insights.” Journal of Official Statistics 17(2):209–26.
Bureau of Transportation Statistics (2006), “Survey Documentation for the Bureau of Transportation Statistics
Omnibus Survey Program (Public Use), November 2006,” Washington, D.C.: Bureau of the Transportation
Statistics.
Cheng, S. (1998), “Who are the Reluctant or Rarely-at-home Respondents?” Survey Methods Center Newsletter 18:811.
Curtin, R., Presser, S., and Singer, E. (2000), “The Effects of Response Rate Changes on the Index of Consumer
Sentiment,” Public Opinion Quarterly 64:413-28.
------- (2005), “Changes in Telephone Survey Nonresponse over the Past Quarter Century,” Public Opinion Quarterly
69:87-98.

De Leeuw, E. and de Heer, W. (2002), “Trends in Household Survey Nonresponse: A longitudinal and International
Comparison,” In Survey Nonresponse, eds. Robert M. Groves, Don A., Dillman, John L. Elting, and Roderick J. A.
Little, pp. 3-26. New York: Wiley.
Dillman, D. A., Elitinge, J. L., Groves, R. M., and Little, J. A. (2002), “Survey Nonresponse in Design, Data
Collection, and Analysis,” In Survey Nonresponse, eds.
Robert M. Groves, Don A. Dillman, John L. Elting, and Roderick J. A. Little, pp. 3-26. New York: Wiley.
Drew, J. H. and Fuller, W. A. (1980), “Modeling Nonresponse in Surveys with Callbacks,” Proceedings of Survey
Research Methods Section, American Statistical Association pp. 639-42.
Dunkelberg, W. and Day, G. (1973), “Nonresponse Bias and Callbacks in Sample Survey,” Journal of Marketing
Research 10:160-68.
Ellis, R. A., Endo, C. M., and Armer, J. M. (1970), “The Use of Potential Nonrespondents for Studying Nonresponse
Bias,” Pacific Sociological Review 13: 103-09.
Etter, J. and Perneger, T. V. (1997), “Analysis of Nonresponse Bias in a Mailed Health Survey,” Journal of Clinical
Epidemiology 50:1123-28.
Filion, F. L. (1976), “Exploring and Correcting Nonresponse Using Follow-Ups of Nonrespondents,” Pacific
Sociological Review 19:401-08.
Fitzgerald, R. and Fuller, L. (1982), “I Hear You Knocking But You Can’t Come in: The Effect of Reluctant
Respondents and Refusers on Sample Survey Estimates,” Sociological Methods and Research 11:3-32.
Gmel, G. (2000), “The Effect of Mode of Data Collection and of Nonresponse on Reported Alcohol Consumption: A
Split-Sample Study in Switzerland,” Addiction 95:123-34.
Groves, R. M. (2006), “Nonresponse Rate and Nonresponse Bias in Household Surveys,” Public Opinion Quarterly
70:646-75.
Groves, R. M., and Wissoker, D. (1999), “Early Nonresponse Studies of the 1997 National Survey of America's
Families,” Assessing the New Federalism Report no. 7 Urban Institute: Washington, DC.
Groves, R. M., and Couper, M. P. (1998), Nonresponse in Household Interview Surveys. New York: Wiley.
Hawkins, D. F. (1975), “Estimation of Nonresponse Bias,” Sociological Methods and Research 3:361-487.
Hilgard, E. R. and Payne, S. L. (1944), “Those Not at Home: Riddle for Pollsters,” Public Opinion Quarterly 8:254261.
Hill, A., Robert, J., Ewings, P., and Gunnell, D. (1997), “Nonresponse Bias in a Lifestyle Survey,” Journal of Public
Health Medicine 19: 203-7.
Johnson, T. P., Cho, Y. I., Campbell, R. T., and Holbrook, A. L., (2006), “Using Community-Level Correlates to
Evaluate Nonresponse Effects in a Telephone Survey,” Public Opinion Quarterly 70:704-19.
Kristal, A. R., White, E., Davis, J. R. et al. (1993), “Effects of Enhanced Calling Efforts on Response Rates, Estimates
of Health Behavior, and Costs in a Telephone Health Survey Using Random-digit Dialing,” Public Health Rep
108:372-9.
Lahaut, V. M. H., Jansen, H. A. M., van de Mheen, D., and Garretsen, H. F. L. (2002), “Nonresponse in a Sample
Survey on Alcohol Consumption,” Alcohol and Alcoholism 37:256-60.
Larson, R. F. and Catton, W. R. (1959), “Can the Mail-Back Bias Contribute to a Study’s Validity?” American
Sociological Review 24:243-245.

Lavrakas, P. J., Bauman, S. L., and Merkle, D. M. (1992), “Refusal Report Forms (RRF), Refusal Conversions, and
Non-response Bias,” Presented at the 47th Annual Meeting of the American Association for Public Opinion
Research, St. Petersburg, Florida, May 15-19.
Lin, I. F. and Schaefer, N. C. (1995), “Using Survey Participants to Estimate the Impact of Nonparticipation,” Public
Opinion Quarterly, 59:236-58.
Lynn, P., Clarke, P., Martin, J., and Sturgis, P. (2002), “The Effects of Extended Interviewer Efforts on Nonresponse
Bias,” In Survey Nonresponse, eds.
Robert M. Groves, Don A. Dillman, John L. Elting, and Roderick J. A. Little, pp. 135-47. New York: Wiley.
Pace, R. C. (1939), “Factors Influencing Questionnaire Returns from Former University Students,” Journal of Applied
Psychology, 23:388-397.
Potthoff, R., Manton, K., and Woodbury, M. (1993), “Correcting for Nonavailability Bias in Surveys by Weighting
Based on Number of Callbacks,” Journal of the American Statistical Association 88:1197-207.
Reid, S. (1942), “Respondents and Nonrespondents to Mail Questionnaires,” Educational Research Bulletin, 21:87-96.
Suchman, E. A. and McCandless, B. (1940), “Who Answers Questionnaires,” Journal of Applied Psychology 24:758769.
Suttleworth, F. K. (1940), “Sampling Errors Involved in Incomplete Returns to Mail Questionnaires,” Psychological
Bulletin, 37:437.
Thomsen, I. B. and Siring, E. (1983), “On the Causes and Effects of Nonresponse: Norwegian Experiences,” in W. G.
Madow and I. Olkin (eds.), Incomplete Data in Sample Surveys, Vol. 3, New York: Academic Press.
Traugott, M. W. (1987), “The Importance of Persistence in Respondent Selection for Pre-election Surveys,” Public
Opinion Quarterly 51:48-57.
Triplett, T., Blair, J., Hamilton, T. et al. (1996), “Initial Cooperators vs. Converted Refusers: Are there Response
Behavior Differences?” In: 1996 Proceedings of the Section on Survey Research Methods, American Statistical
Association. Alexandria, VA: American Statistical Association, 1996:1038-41.
Voigt, L. F., Koepsell, T. D., and Darling, J. R. (2003), “Characteristics of Telephone Survey Respondents According
to Willingness to Participate,” American Journal of Epidemiology 157:66-73.