Monetary Incentive Literature

An Experiment in Monetary Incentives
M. Berlin, L. Mohadjer, J. Waksberg, A. Kolstad, I. Kirsch, D. Rock, and K. Yamamoto

Introduction
The National Adult Literacy Survey (NALS) is a
household survey designed to measure and report on the
nature and extent of literacy problems facing the
population of the United States. Some 15,000 adults
aged 16 and older (plus an additional 12,000 adults from
the State Adult Literacy Surveys) responded to a onehour survey that consisted of a 15-minute background
questionnaire and a 45-minute literacy exercise
assessing their prose, document, and quantitative
literacy skills.
In accordance with the contract between the
Educational Testing Service in cooperation with
Westat, Inc., and the U. S. Department of Education, a
field test of some 2,000 adults aged 16 and older was
conducted in a sample of 16 Primary Sampling Units
(PSUs) drawn from the contiguous 48 states. One of
the purposes of the field test was to evaluate the impact
of incentives on response rates, performance, and
survey costs.
After presenting a brief review of the literature
on incentive experiments (Section 2), this paper
provides a detailed description of the N ALS field-test
design (Section 3), along with analyses of the field-test
results (Section 4).
1.

2.

Literature Review
The widespread use of survey ~ h
today as a
means of gathering information makes its cost a matter
of considerable interest (Reingen & Kernan, 1977).
Because much of a survey's cost goes into achieving
high resimnse rates, cow~rned researchers have explored
a variety of response inducement techniques over the
past 25 years. In one extensive effort to determine
which methods of response rate improvement are most
effective, Kanuk & Berenson (1975) examined over 75
articles that addressed increasing mail survey response
rates. They found that follow-up contact and the use of
monetary incentives were the only two methodological
procedures that had any empirical impact on response
rates.
Much of the research has found that the use of
monetary incentives does increase response rates in
surveys. In suplx~ of this f'mding, Armstrong (1975)
reviewed 18 studies from 14 different researchers and
concluded that not only do monetary incentives in mail
surveys yield large increases in response rates but that
the larger the incentive, the greater the increase in
reslxmse rate.
In a review of literature focusing on improving
survey response rates, Baxter, et al (1984) discussed the
theory that most respondents need a reason for taking
the time to participate in a survey. Baxter's study
indicates that for all respondents there is some small

393

amount of money which functions as a symbolic
reward that g e n ~
participation. They note, however,
that part of the inconsistency in the research on
incentives and response rates is that the financial
standing of respondents determines the range of
amounts that serve as token rewards.
Several reports on personal interview studies and
the effect of monetary incentives on response rates are
consistent with the mail survey results (Gunn &
Rhodes, 1981).
The only study that has not shown improvement
on response rate with incentives is a 1972-73
household survey conducted by the Census Bureau for
the Bureau of Labor Statistics (Walsh, 1977).
However, there were design and operational problems,
and the Census Bureau noted that "results might have
been different if the experiment had been conducted after
resolution of certain of these problems."

3.

The NALS Field-Test Design
The NALS field test was conducted in a sample
of 16 Primary Sampling Units (PSUs), consisting of
counties or groups of counties representing the
contiguous 48 states. The PSUs were selected in such a
way that satisfied a Latin Square design based on key
variables thought to be related to response rates. The
variables included region of the country, urbanicity,
race/ethnicity, and average income/education level of
persons residing in the selected PSUs. On average,
about 21 segments (consisting of census blocks) were
selected within each PSU, with about 8 households
selected in each segment.
Incentive levels of $0, $20, and $35 to be
evaluated in the field test were agreed upon by NCES
and OMB. Incentives were randomly assigned to
segments so that each incentive group had about the
same number of cases and there were about the same
number of incentive groups represented at the PSU and
census region levels.
Approximately two interviewers were assigned
to each of the PSUs to complete the field work. Each
of 38 interviewers was assigned an approximately equal
number of segments in the three payment groups.
Interviewers were instructed to introduce the incentive
after the house~Id composition was determined and the
eligible respondent(s) selected. Respondents were paid
by check at the completion of the exercise booklet.
Only respondents who completed the background
questionnaire and agreed to take the exercise booklet
were given the incentive check. In order to compare the
costs of the three incentive levels (as well as response
rates and other measures of quality of the results),
interviewers were required to record time spent and
expenses incurred by segment each week.

Analyses of the NALS Field Test
The analyses of the field-test data showed that
payments of incentives significantly improved the
outcomes of the survey. The major areas showing
signifw.ant improvement by providing incentives to the
reslmndents were the following:
• Response Rates. There were significant
increases in response rates for the background
questionnaire and exercise booklets (iw.entives were not
introduced with the screener questionnaire) as a result of
offering incentives to the respondents. Given the
sample size, no statistically significant difference in
response rates was detected between the $20 and $35
incentive groups.
• Representation of the Target Population.
Analyses showed that the incentives were most
effective in improving response rates for people with
low educational attainment and minority populations
who are frequently underrepresented in national
household surveys (a discussion of the issues related to
undercoverage in household surveys is provide~ later).
As a result, the incentives provided a b e t t e r
representation of the general population and improved
the representation of subgroups, such as the Black and
Hispanic populations, that are of special interest to
NALS.
• Relationship between Incentive Levels, SelfSelection, and Performance. The accuracy of the
estimates of adults' literacy proficiencies depends on
gaining the cooperation of the majority of eligible
respondents to take the literacy exercise. A significant
number of refusals within any one of the three
incentive levels will not bias the results only if the
refusals do not differ in any relevant or systematic way
from the respondents who complete the exercises in the
remaining incentive levels. Results of the field test
show that if no incentive payment is offered to eligible
respondents, the main assessment will substantially
increase bias in estimates of the population's
literacy level. This increase in the bias is likely to
result from self-selection factors occurring in the $0
incentive group that lead to a nonrep~sentative sample.
If no incentive is used in the main assessment, these
self-selection factors will result in an overestimate of
the literacy levels in the United States for both the total
population and major subgroups. Possible remedies
would be the use of post-suatification proc~ures. It is
our opinion, however, that these Im3cedures would not
completely eliminate the apparent bias.
• Survey Costs. A cost analysis of the field-test
experiment showed a reduction in interviewing costs
when incentives were given to the respondents. The
cost per completed interview (including the cost of the
incentive) for respondents in the $20 incentive group
was lower than in the $0 and $35 incentive groups.
The remainder of this paper provides a more indepth discussion of the results of the field test with
respect to the impact of incentives. The response rates

4.

394

for different incentive levels are presented in Section
4.1. Section 4.2 contains a discussion of the
improvements in the representation of the target
population when incentives are paid to respondents.
The relationship between incentive levels, selfselection, and performance is presented in Section 4.3.
The effect on costs is examined in Section 4.4. Finally,
Section 5 presents the summary and conclusions of the
analyses.
4.1 Response Rates
The survey included three insmanents: screener,
background questionnaire, and exercise booklet. The
screener coUected household level data for the purpose
of selecting an eligible respondenL If the household
had been assigned an incentive, the selected reslx~dent
was told that an incentive would be paid upon the
completion of the survey instruments. The background
questionnaire collected information in six areas:
demographic data, language background, education,
political and social participation, labor force
participation, and literacy activities. The exercise
booklet consisted of three 15-minute sections of prose,
document and quantitative tasks.
There were 2,774 households in the sample. Out
of these 2,774 households, 336 (12.1 percent) units
were found either to be vacant or not satisfying the
definitions of a dwelling unit at the time of screening.
As a result, 2,438 households were found to be
eligible, of which 2,155 (88.4 percent) completed the
screener. Out of 283 nonresponding households, 152
(6.2 percent) refused to participate in the study, and 131
(5.4 percent) did not complete the screener for other
reasofls.

In households with completed screeners, one
person was randomly selected if there were three or
fewer eligible people in the households. Two persons
were selected in households with four or more eligible
people. After the respondent(s) was selected, the
background questionnaire was administered.
A total of 2,288 eligible respondents were
selected for the b a c ~ u n d interview; of that total, 412
did not complete the background questionnaire. Of the
412 who did not complete the background
questionnaire, 258 (62.6 percent) were refusals, 39 (9.5
percent) had some type of mental or physical disability,
and the remainder were nonrespondents for other reasons
(such as broken appointments, language barrier, and so
for,h).
The response rate analyses indicated a
statistically significant improvement in the background
questionnaire response rates for respondents given
incentives. The response rate incre.asexl by about 4
percent when an incentive of $20 was paid to the
respondents. There were no statistically significant
improvements in response rate when the incentive was
incte,ased from $20 to $35.
All 1,876 persons who completed the
background questionnaire were asked to take an

exercise. Among the different incentive levels, the
same pattern of respcmse rates as observed for the
background questionnaire was observed for the exercise
booklet. The analysis showed that there was a
significant improvement (5 percent) in exercise
response rates for respondents given incentives. There
were, however, no statistically significant differences
between the $20 and $35 incentives. Whe~ the $20 and
$35 incentive level groups were combined, there was a
net gain of 9 percent over the $0 incentive group in the
combined response rate for the background
questionnaire and the literacy exercise. The following
is a summary of response rates by incentive level.

the minority populations also increased by about 10
percent as a result of offering significantly $20
incemives. This increase in response rate was also
statistically significant. Thus, the overall effect of the
$20 incentive was to add about 20 percent to the
response rate for minorities. Given the sample sizes,
there were no sw~fically significant differences in
response rates between the $20 and $35 incentives.

Summary )f response rates by incentive levels ,.
Incentive
$0
$20
$35
Screener
87.4%
87.7%
90.0%
Backgmtmd . 78.6%
82.7%
84.4%
Exercise
92.8%
97.9%
98.0%
Overall
63.75%
71.0%
74.4%
|l|

i

Almost all of the nonrespondents were persons
who refused to take the literacy exercises. The
proportion of those who refused to do the exercise
booklet was significantly higher for the $0 incentive
group than for the $20 or $35 incentive groups.
4.2 Representation of the Target
Population
When a monetary incentive was paid, a
disproportionate share of the increase in response rates
occurred in subgroups of the population that are of
special interest to literacy studies, that is, minorities
and undereducateA persons. These two groups currexttly
account for over 20 percent of the United States
population.
It is particularly important to achieve good
response rates for minorities so as to compensate
partially for the undercount that almost always occurs
in household surveys. The evaluations carried out of the
decennial censuses indicate that there has been a
recurrent undercount of about two to three percent of
the population. Furthermore, since at least 1950,
coverage of the Black population in censuses has
lower than that of the White population, with the
proportion of uncounted Black males higher than other
major demographic subgroups. Recent studies of
undercounts have shown that the coverage for the
Hispanic population may even be a little lower than
that for the Black population. Sample surveys usually
do not even do as well as censuses. Consequently, low
response rates resulting from not offering an incentive
would intensify even further the potential coverage
bias.
There was a statistically significant increase of
about nine percent in the background questionnaire
response rate for the minority population (Black and
Hispanic populations combined) as a result of
providing $20 incentive. The exercise response rates for

395

Persons with less educational attainment made
up an important subgroup to study from a policy
perspective. The experiment showed a large increase in
exercise response rates in the $20 and $35 incentive
groups for persons with no high school diploma. The
same pattern exists for those with only a high school
diploma. There was a significant increase of about 8
percent in the exercise response rate for persons both
with no high school diploma and with only a high
school diploma when a $20 incentive was paid to the
respondents. The increase m response rates for persons
with some college or a college degree was not found to
be significant.
Finally, the incentive experiment showed a
significant increase in response rates for persons aged
16 to 64 when incentives were paid to the respondents.
There was an increase of about 8 percent in background
questionnaire response rate for persons aged 16 to 64
when $20 incentives were paid and an increase of about
4 percent was observed in exercise response rate. Both
the questionnaire and exercise response rates increases
were proven to be statistically significant for the 16-to64-year old Ix)pulation.
4.3 Relationship between Incentive Level,
Self-Selection, and Performance
This section evaluates the impact of incentives
on the demonstrated distribution of proficiency scores.
Through examination of the distributions of estimated
literacy proficiency scores, the field-test data provide a
means for discovering whether or not the groups
agreeing to complete the literacy tasks represent
individuals of similar demonstrated literacy proficiency.
That is, is there evidence that individuals of higher (or
lower) literacy Woficiency are more likely to participate
under a given incentive condition?
Before discussing the results of the analyses, it
may be helpful to describe briefly the assessment
booklets (the combination of literacy or cognitive
tasks) and the literacy score estimates derived from the
field test. Some 100 literacy tasks were developed and
assembled into nine discrete blocks, each of which was
expected to require 15 minutes of administration time.
Each block contained approximately the same number
of questions contributing to each of the three literacy
scales. The nine blocks of tasks were then assembled
into nine booklets, each requiting a total of
approximately 45 minutes of administration time. The
booklets were configured so that the same three blocks
appeared together in three booklets, with each block
falling in each position --- that is, as the first, middle,

and last block of a ~ e t
TI~ item response theory
(lilT) scaling pnr.edme employed to ~ t e
literacy
proficiency scores allowed one to put all scores on a
scale, even du~gh gnmps of individuals took different
sets of tasks (la3~ 1980). The IRT scale defined for the
field test had a mean of 50 and a standard deviation of
10 for the proficiency distribution of the document
literacy scales. One key question is wheater or not the
individuals who did cooperate under the $0 incentive
level are different with respect to literacy proficiency
from those who cooperated under the other two
incentive levels. To address this question, a two-way
analysis of variance was conducted where the
independent variables were test Ix)oklets (three groups
of booklets) and three levels of incentives - - $13, $20,
and $35. The dependent variables were item response
theory (IRT) scale scores. The analysis of variance
showed that incentive level had a significant impact on
performance. The next important question investigate
dealt with the nature of the significant impact that
incentive level had on literacy performance.

Table 1 ~ t s
literacy performance on the, document,
literacy scale by incentive levels, as well as by selected
demographics c r o ~
with incentive levels. These
cross-classifications indicate that the significant
analysis of variance results can be accounted for by
performance level differences betwom the $0 incentive
group and the remaining two incentive levels. The total
mean literacy scores for the $0 incentive level are
significantly higher than the correslxmdin~ means for
the $20 and $35 incentive levels for all three scales.
There is no significant difference between the total
mean literacy scores for the $20 and $35 incentive
levels on any of the three literacy scales. Further
inspection of the data indicates that the performance
level difference in favor of the $0 incentive individuals
is about 20 percent of a standard deviation on each of
the three scales. Typically, in the education evaluation
literature, a difference in group mean performance of
greater than one-tenth of a standard deviation - - "a
small but nontrivial difference" n is judged to be of
Wactical educational significance (Cohen, 1988).

Table 1. Proficienc, means and standarddeviationsby incentive and scale
Total
$0
$20
N
Mean
N
Mean
N
Mean
Total
1707
50.0
495
50.8
49.6
563
Gender
Male
775
50.4
225
51.4
271
49.9
Female
917
49.6
264
50.4
290
49.3
Ethnicity
White
1370
51.2
441
405
51.7
50.9
Black
171
42.7
39
44.4
60
43.6
Hispanic
124
46.4
43
49.2
45
45.5
Asian
26
52.2
11
5
47.6
52.6
Amerind
11
46.7
4
44.9
1
30.6
Other
3
49.2
2
52.4
1
42.8
Education
InHS
68
50.1
20
49.5
21
50.8
File Typeapplication/pdfFile Title1992: An Experiment in Monetary IncentivesFile Modified2008-12-03File Created2002-07-29