MilCohort Smallpox Vaccination Comparison November07 Human Vaccines

[Human Vaccines 3:6, 245-251, November/December 2007]; ©2007 Landes Bioscience

Research Paper

Smallpox Vaccination
Comparison of Self-Reported and Electronic Vaccine Records
in the Millennium Cohort Study
Cynthia A. LeardMann1,*
Besa Smith1
Tyler C. Smith1
Timothy S. Wells2
Margaret A. K. Ryan1
for the Millennium Cohort Study Team
1Department of Defense Center for Deployment Health Research; Naval Health
Research Center; San Diego, California USA
2Air Force Research Laboratory; Wright-Patterson Air Force Base, Ohio USA

*Correspondence to: Cynthia LeardMann; Department of Defense; Center for
Deployment Health Research; Naval Health Research Center; P.O. Box 85122;
San Diego, California 92186 USA; Tel.: 619.553.7594; Fax: 619.553.7601;
Email: [email protected]
Original manuscript submitted: 04/23/07
Manuscript accepted: 06/14/07
Previously published online as a Human Vaccines E-publication:
http://www.landesbioscience.com/journals/vaccines/article/4529

Key words
smallpox vaccine, questionnaires, military
medicine, quality of life, validation studies

In December 2002, the US Government implemented policy to immunize health
workers, first responders and military personnel against smallpox in preparation for
a possible bioterrorist attack. Self‑reported vaccination data are commonly used in epidemiologic research and may be used to determine vaccination status in a public health
emergency. To establish a measure of reliability, the agreement between self‑reported
smallpox vaccination and electronic vaccination records was examined using data from
the Millennium Cohort Study. Descriptive measures and a kappa statistic were calculated
for data from 54,066 Millennium Cohort Study participants. Multivariable modeling
adjusting for potential confounders was used to investigate vaccination agreement status
and health metrics, as measured by the Short Form 36‑Item Health Survey for Veterans
(SF‑36V) and hospitalization data. Substantial agreement (k = 0.62) was found between
self‑report and electronic recording of smallpox vaccination. Of all participants with
an electronic record of smallpox vaccination, 90% self‑reported being vaccinated;
and of all participants with no electronic record of vaccination, 82% self‑reported not
receiving a vaccination. There was no significant difference in hospitalization experience
prior to questionnaire completion between vaccinated and unvaccinated participants.
While overall scores on the SF‑36V suggested a healthy population, participants
whose self‑reported vaccination status did not match electronic records had slightly
lower adjusted mean scores for some scales. These results indicate strong reliability
in self‑reported smallpox vaccination and also suggest that discordant reporting
of smallpox vaccination is not associated with substantial differences in health among
Millennium Cohort participants.

INTRODUCTION

Abbreviations
CI
DMDC

DoD

GWOT
κ
SF-36V

Abstract

confidence interval
Defense Manpower Data
Center
US Department of
Defense
Global War on Terrorism
kappa statistic
Medical Outcomes Study
Short Form 36-item
Health Survey for Veterans

Acknowledgements
See page 250.

www.landesbioscience.com

The World Health Organization announced in 1980 that naturally occurring smallpox
had been eradicated worldwide following the last case reported in Somalia in 1977.1,2
Once the risk of natural infection had disappeared, routine vaccination was discontinued.
Laboratories with existing smallpox (variola virus) were requested to destroy the virus
or turn their stocks over to one of two collaborating centers.2,3 It has been suggested,
however, that the variola virus may have been retained for use as a biological weapon.4
In the wake of the September 11, 2001 terrorist attacks, the US Government began
to immunize health workers, first responders and military personnel against smallpox
in preparation for a possible bioterrorist attack.3 From December 2002–January 2005,
nearly 40,000 health care workers and first responders were inoculated against smallpox
as well as more than 730,000 US military personnel.5
When possible, establishing a measure of reliability in self‑reported data will yield
insight into potential strengths and limitations of questionnaire data. Most adult vaccine
studies have indicated that self‑report is a moderately reliable method to determine
an individual’s vaccination status,6‑13 but no previous study to date has investigated
the accuracy of self‑reported smallpox vaccination. Previous studies have found that
self‑report of adult influenza and pneumococcal vaccination is highly sensitive (70–100%)
but less specific (22–89%).6,7,10,13 Using limited anthrax vaccine data, the sensitivity
of self‑reported anthrax vaccination among 1991 Gulf War veterans was 73.9%.8 Among
Millennium Cohort Study members, greater than substantial agreement (k = 0.80)
was found between self‑report and electronic anthrax vaccination status.11 Two previous
studies found smallpox vaccinees were accurately able to self‑report their reaction to the
smallpox vaccination.14,15 Using data from the Millennium Cohort Study, the purpose
of this study was to investigate agreement between self‑reported smallpox vaccination
Human Vaccines

245

Validation of Smallpox Vaccination and Measures of Health

and electronic vaccination records in a large, population‑based
military cohort. Further, this study examines the variation in
vaccination data concordance as it pertains to subjective and
objective measures of physical and mental health.

computerized databases of standardized discharge diagnoses for
hospitalizations within the Military Health System and for
hospitalizations billed to the Department of Defense by nonmilitary facilities. These databases contain hospitalization summaries
including dates of admission and discharge, and up to eight
individual discharge diagnoses for each encounter, which are
Methods
uniformly coded across US military services. The Medical
Study population and data sources. The Millennium Cohort Outcomes Study Short Form 36‑Item Health Survey for Veterans
Study was launched in 2001 to gather and evaluate population‑based (SF‑36V), a modified version of the MOS Short Form 36 (SF‑36)
data on behavioral and occupational risk factors related to military that is contained within the Millennium Cohort questionnaire,
service that may be associated with adverse health outcomes.16,17 was used to compare differences in physical and mental
A randomly selected group from all US military personnel serving health.18‑22 Like the SF‑36, the SF‑36V uses standardized scoring
in 2000 was invited to participate in the first panel. Those who algorithms to assess eight health scales: physical functioning,
had been deployed to Southwest Asia, Bosnia or Kosovo role limitations caused by physical problems, bodily pain, general
(January 1, 1998–September 1, 2000), members of the Reserve or health, vitality, social functioning, role limitations caused by
National Guard and women were oversampled to ensure sufficient emotional problems and mental health. Higher scores are associated
power to detect differences in these smaller population subgroups. with better health status. Unique identifiers were used to link
Of the 77,047 enrolled in this first panel, 55,021 (71.4%) the immunization, demographic and hospitalization data to the
completed the follow‑up questionnaire between June 2004 and Millennium Cohort participants.
February 2006. The population for the current study included
Statistical analysis. A descriptive investigation was completed to
Millennium Cohort Study participants with complete demo- compare self‑reported vaccination status and electronic vaccination
graphic and military‑specific data who consented at baseline, records. The percentage of disagreement between self‑reported vacciparticipated in the follow‑up, and answered the smallpox vaccine nation and electronic documentation of smallpox vaccination was
question on the follow‑up questionnaire.
reported. To measure the degree of nonrandom agreement between
The Defense Manpower Data Center (DMDC) provided demo- self‑reported smallpox vaccination and electronic smallpox vaccine
graphic and military‑specific data from electronic personnel files, records the kappa statistic was used.23 A k between 0.8–1.0 was
including gender, birth date, highest education level, marital status, defined as “greater than substantial agreement,” between 0.6–0.8
race/ethnicity, deployment experience (no deployment experience; as “substantial agreement,” between 0.4–0.6 as “moderate agreeGulf War, Bosnia, Kosovo or Southwest Asia deployment experi- ment,” between 0.2–0.4 as “fair agreement,” and between 0.0–0.2
ence before the Global War on Terrorism (GWOT); or deployment as “slight or poor agreement.”24
experience in support of GWOT between 2001 and 2006),
To assess associations between prior hospitalization and
pay grade, service component (active duty or Reserve/National vaccination concordance status multivariable logistic regression was
Guard), service branch (Army, Air Force, Navy, Coast Guard used, adjusting for all demographic and military‑related variables.
or Marine Corps) and primary military occupations.
Analysis of variance was performed to investigate the association
Smallpox vaccination data. The following question was included between self‑reported health status (SF‑36V) and vaccination
on the 2004–2006 questionnaire to measure smallpox vaccination, concordance, adjusting for all demographic and military‑related
“In the past three years have you received the smallpox vaccine?” variables. Vaccination concordance was divided into four categories:
The questionnaire did not include information about the (1) participants who reported not receiving a smallpox vaccination
vaccine’s administration or skin reaction at the injection site. with concurrence from the electronic records, (2) participants who
Electronic smallpox vaccination data were obtained from the Defense reported receiving a smallpox vaccination but electronic records did
Enrollment and Eligibility Reporting System at DMDC which not reflect vaccination, (3) participants who reported not receiving
maintains an electronic database of all vaccinations given to military a smallpox vaccination but electronic records reflected vaccination
service members. A participant was classified as vaccinated in the and (4) participants who reported receiving a smallpox vaccination
electronic records if a smallpox vaccination was documented in the with concurrence from the electronic records.
three years prior to completing the follow‑up questionnaire.
Regression diagnostics, including examining covariates for multiHealth metrics. The baseline and follow‑up questionnaires collinearity and goodness of fit test, were performed. All data analyses
of the Millennium Cohort Study include questions to assess physical were completed using SAS Version 9.1.3 (SAS Institute, Inc., Cary,
health, mental health, and other health outcomes. Differences North Carolina).
between vaccination concordance groups were examined as they
pertained to prior hospitalization and subjective health metrics.
Results
Inpatient data were used to investigate differences in objective
measures of health using hospitalization as the outcome of interest
This study included 54,066 Millennium Cohort participants
among active‑duty service members. Participants were classified who completed the follow‑up questionnaire between June 2004–
as hospitalized if, in the 12 months prior to completing the survey, February 2006, answered the smallpox vaccination question,
they were hospitalized for any cause, excluding the following and had complete demographic and military‑specific data.
International Classification of Diseases, Ninth Revision, Clinical The majority of participants self‑reported they had not received
Modification diagnosis codes: complications of pregnancy, child- a smallpox vaccination in the three years prior to completing the
birth, and the puerperium (630–77), congenital anomalies questionnaire (n = 34,859, 64.5%). The electronic vaccination
(740–59), and certain conditions originating in the perinatal records confirmed 33,614 (96.4%) of these participants had not
period (760–79). Hospitalization data were obtained from the received a smallpox vaccination. A smaller percentage of participants
246

Human Vaccines

2007; Vol. 3 Issue 6

Validation of Smallpox Vaccination and Measures of Health

Table 1

Characteristics of Millennium Cohort Study participants by smallpox vaccination status

Smallpox vaccination status
Discordant † Discordant †

Study
Concordant †
self-report
electronic

sample
unvaccinated
vaccinated
vaccinated
N = 54,066 N = 33,614

Concordant †
vaccinated

n = 7,537 N = 1,245 N = 11,670

p-value‡

Characteristic
%*
n (%)
n (%)
n (%)
n (%)
Gender
<.0001
Male
Female

73.4
26.6

23,477 (69.8)
10,137 (30.2)

5,935 (78.7)
1,602 (21.3)

922 (74.1)
323 (25.9)

9,331 (80.0)
2,339 (20.0)

Birth Year
Before 1960
1960–1969
1970–1979
1980 or later

24.5
40.6
30.8
4.1

9,072 (27.0)
13,497 (40.2)
9,896 (29.4)
1,149 (3.4)

2,123
3,144
1,923
347

(28.2)
(41.7)
(25.5)
(4.6)

204
559
429
53

(16.4)
(44.9)
(34.5)
(4.3)

1,849
4,723
4,408
690

Education
High school or less
Some college
Bachelor’s degree
Advanced degree

45.6
17.8
22.1
14.6

13,569 (40.4)
6,942 (20.7)
7,603 (22.6)
5,500 (16.4)

3,780 (50.2)
1,104 (14.7)
1,678 (22.3)
975 (12.9)

675
207
227
136

(54.2)
(16.6)
(18.2)
(10.9)

6,621
1,367
2,428
1,254

26.7
73.3

8,871 (26.4)
24,743 (73.6)

2,038 (27.0)
5,499 (73.0)

301 (24.2)
944 (75.8)

71.0
12.1
16.9

24,209 (72.0)
4,031 (12.0)
5,374 (16.0)

5,178 (68.7)
1,032 (13.7)
1,327 (17.6)

829 (66.6)
212 (17.0)
204 (16.4)

No deployment
46.3
Pre-GWOT deployment
23.1
2001–2006 GWOT deployment
15.3
Both pre-GWOT and GWOT deployment 15.2

19,762 (58.8)
9,875 (29.4)
1,835 (5.5)
2,142 (6.4)

3,635 (48.2)
1,800 (23.9)
1,135 (15.1)
967 (12.8)

257
142
397
449

(20.6)
(11.4)
(31.9)
(36.1)

1,398
673
4,923
4,676

70.8
29.2

23,528 (70.0)
10,086 (30.0)

5,434 (72.1)
2,103 (27.9)

946 (76.0)
299 (24.0)

53.4
46.6

18,560 (55.2)
15,054 (44.8)

4,266 (56.6)
3,271 (43.4)

476 (38.2)
769 (61.8)

Army
Air Force
Navy/Coast Guard
Marine Corps

47.7
30.3
18.0
4.0

15,035 (44.7)
10,540 (31.4)
6,752 (20.1)
1,287 (3.8)

3,624
1,673
1,823
417

(48.1)
(22.2)
(24.2)
(5.5)

554
511
151
29

(44.5)
(41.0)
(12.1)
(2.3)

6,577
3,644
1,017
432

19.2
11.4
22.9
46.5

5,977 (17.8)
4,106 (12.2)
8,449 (25.1)
15,082 (44.9)

1,514
526
1,820
3,677

(20.1)
(7.0)
(24.2)
(48.8)

235
171
236
603

(18.9)
(13.7)
(19.0)
(48.4)

2,663
1,352
1,879
5,776

<.0001

(56.4)
(31.2)
(8.7)
(3.7)

Occupational category
Combat specialists
Health care specialists
Functional support
Others

<.0001

5,570 (47.7)
6,100 (52.3)

Branch of service

<.0001

8,374 (71.8)
3,296 (28.2)

Service component
Reserve/Guard
Active duty

<.0001

(12.0)
(5.8)
(42.2)
(40.1)

Military rank
Enlisted
Officer

<.0001

8,185 (70.1)
1,279 (11.0)
2,206 (18.9)

Deployment experience §

0.009

3,228 (27.7)
8,442 (72.3)

Race/ethnicity
White non-Hispanic
Black non-Hispanic
Other

<.0001

(56.7)
(11.7)
(20.8)
(10.8)

Marital status
Not married
Married

<.0001

(15.8)
(40.5)
(37.8)
(5.9)

<.0001

(22.8)
(12.0)
(16.1)
(49.5)

*Percentages rounded and may not sum to 100. †Concordant unvaccinated, both self-report and electronic database reflect no vaccination; Discordant self-report vaccinated, self-reported vaccination but electronic
database reflects no vaccination; Discordant electronic vaccinated, self-reported no vaccination but electronic database reflects vaccination; Concordant vaccinated, both self-report and electronic database reflect
vaccination. ‡Pearson chi-square p value examining unadjusted associations between smallpox vaccination status and individual characteristics. §Pre-GWOT deployment, deployed to the 1991 Gulf War or to Bosnia, Kosovo
or Southwest Asia between January 1, 1998, and September 30, 2000; 2001–2006 GWOT deployment, deployed in support of the Global War on Terrorism before completion of follow-up survey.

reported receiving a smallpox vaccination (n = 19,207, 35.5%).
Of these participants, 11,670 (60.8%) had an electronic record
indicating vaccination. Among all participants, 8,782 (16.2%) had
discordant results. Most participants with discordant results reported
receiving a vaccination even though electronic records indicated
www.landesbioscience.com

no vaccination (n = 7,537; 13.9%). There were 1,245 (2.3%)
with electronic documentation of smallpox vaccination who reported
not receiving the vaccination.
The demographic characteristics of these four vaccination
groups are compared in Table 1. A greater proportion of those

Human Vaccines

247

Validation of Smallpox Vaccination and Measures of Health

After adjustment for gender, age, education, marital status,
race/ethnicity, deployment experience, pay grade, service branch and
occupation, hospitalization rates for any cause in the year prior
to survey completion were not significantly different between the

Electronic record
vaccination groups among active‑duty service members (Table 3).
Vaccine No vaccine
The SF‑36V mean scores for the eight scales were adjusted
Self-report
(N = 12,915)
(N = 41,151)
for gender, age, education, marital status, race/ethnicity, deployVaccine
ment experience, pay grade, service component, service branch
(N = 19,207)
11,670
7,537
and occupation (Table 3). Overall, the SF‑36V adjusted mean scores
No vaccine
were relatively high for each vaccination group, indicating good
(N = 34,859)
1,245
33,614
health. However, significantly lower adjusted mean scores for social
*Agreement measures for self-report are as follows: k = 0.62; 90.4% of participants with electronic functioning and role limitations caused by physical and emotional
documentation of smallpox vaccination self-reported being vaccinated; 81.7% of participants with problems were found among participants whose electronic records
no electronic record of vaccination self-reported not receiving a vaccination.
reflected no vaccination but who self‑reported receiving the smallpox
vaccination when compared to groups with concordant status.
who had an electronically confirmed self‑reported vaccination
The adjusted mean score for vitality was slightly but significantly
(concordant vaccinated) were male, younger, less educated, deployed
lower among participants who reported they did not receive
in support of GWOT, active duty, Army service members and
a vaccination but whose electronic records reflected a vaccination.
combat specialists than those who reported not receiving a vacciAdditionally, the adjusted mean score for physical functioning
nation, which was confirmed by the electronic vaccination record and general health were slightly but significantly higher among
(concordant unvaccinated). A larger percentage of participants the concordant vaccinated group compared to the concordant
who self‑reported vaccination with no confirmation by electronic unvaccinated group.
record (discordant self‑reported vaccinated) were older, not deployed
in support of GWOT, Reserve/National Guard, in the Navy/Coast
Guard or Marine Corps and working in functional support roles than Discussion
were concordant vaccinated participants.
This study demonstrated that self‑reported smallpox vaccination
Of all participants with electronic documentation of smallpox records are consistent with electronic military vaccination records,
vaccination, 90% self‑reported being vaccinated; and of all partici- with 84% of participants having identical responses from the two
pants with no electronic record of vaccination, 82% self‑reported sources. Agreement levels,24 as demostrated by the kappa statistic,
not receiving a vaccination (Table 2). The overall k statistic indicated suggest substantial agreement between electronic and self‑reported
substantial agreement (k = 0.62). While most of the k statistics smallpox vaccination (k = 0.62). While, to our knowedge,
calculated for the categories of demographic characteristics indi- no previous studies have investigated the validity of self‑
cated substantial agreement (k = 0.6–0.8), participants who were reported smallpox vaccination, previous studies have examined
born before 1960 (k = 0.51), those without GWOT deployment other vaccinations given to adults. A slightly higher agreement
experience (k = 0.35), Navy/Coast Guard service members (k = 0.41) (k = 0.69) was found between self report and medical records for
and Marines (k = 0.53) had lower agreement scores.
pneumococcal vaccination among Australian adults.6 Two other
Table 2

Agreement* between self-reported smallpox
vaccination and electronic records

Table 3

Adjusted odds of prior hospitalization and adjusted means of SF-36V health scores
among Millennium Cohort participants* by smallpox vaccination status
Smallpox vaccination status
Concordant† Discordant† Discordant†
Concordant†
unvaccinated
self-report
electronic
vaccinated

vaccinated
vaccinated
OR (95% CI)‡
OR (95% CI)
OR (95% CI) OR (95% CI)

Health outcome

Any-cause hospitalization§
SF-36V¶

Physical functioning
Role physical
Bodily pain
General health
Social functioning
Role emotional
Mental health
Vitality

1.00

Mean

92.71
93.61
76.31,2
76.91
88.01
94.01
78.61
58.81,2

1.19 (0.97–1.47)

Mean

92.41
92.82
75.62
76.61
86.52
93.22
77.72
58.42

0.88 (0.57–1.35)

Mean

92.01
93.71,2
74.91,2
75.41
87.51,2
93.91,2
77.71,2
56.63

0.88 (0.71–1.10)
Mean
93.52
94.21
76.61
78.22
88.31
94.21
78.41,2
59.41

*Participants whose SF-36V component could not be scored due to insufficient questionnaire responses were removed from analyses (n varies by component; maximum removed: 446). †Concordant
unvaccinated, both self-report and electronic database reflect no vaccination (reference group); Discordant self-report vaccinated, self-reported vaccination but electronic database reflects no vaccination; Discordant electronic vaccinated, self-reported no vaccination but electronic database reflects vaccination; Concordant vaccinated, both self-report and electronic database reflect vaccination.‡ Odds
ratio (OR), 95% confidence interval (CI), and mean are adjusted for gender, age, education, marital status, race/ethnicity, deployment experience, pay grade, service component (SF-36V only), service
branch, and occupation.§ Hospitalization analysis includes active-duty service members only and reflects odds of any-cause admission in the year prior to survey response.¶ SF-36V, Medical Outcomes
Study Short Form 36-Item Health Survey for Veterans. 1-3 Different numbers indicate vaccination status scores that are significantly different from each other (p < 0.05) using Scheffe’s adjustment
for multiple comparisons.

248

Human Vaccines

2007; Vol. 3 Issue 6

Validation of Smallpox Vaccination and Measures of Health

studies, however, conducted among elderly Department of Veterans
Affairs patients have found lower agreement (k = 0.18 and 0.42)
between self‑report and medical record documentation for pneumococcal vaccination.7,10 Among Millennium Cohort participants,
agreement between self‑report and electronic records for anthrax
vaccination has been reported higher (k = 0.80) than the current
study found for smallpox vaccination.11
Consistent with previous research, a higher percentage of participants with electronic documentation recalled receiving a vaccination
(90%) than participants with no electronic record who reported
not receiving a vaccination (82%).7,10,11,13 The finding of a lower
agreement compared with anthrax vaccination in the Millennium
Cohort is interesting. Self‑reported vaccinations administered
annually or in succession of doses, such as influenza and anthrax,
tend to be more reliably recalled than vaccinations administered in
a single dose or infrequently, like pneumococcal and smallpox.6,7,10
Primary vaccination against anthrax includes a six‑dose series followed
by an annual booster, whereas smallpox vaccination consists of one
dose rarely followed by a booster.25,26 Even though it is administered in one dose, the smallpox vaccination is more distinct than
other vaccinations, with vaccinees developing a skin reaction at the
injection site that lasts up to three weeks and usually leaves a permanent scar.1 Therefore, it seems likely that vaccinees would remember
this vaccination. The largest source of disagreement (86%), however,
is from those who stated they received a smallpox vaccination but
did not have electronic documentation. Furthermore, the smallpox
question on the survey asks if the vaccine was received in the
past three years, whereas the anthrax question asks if the vaccine
has ever been received. Since the smallpox vaccine was routinely
administered to children in the United States until 1971 and to
military recruits until 1990, it is possible that service members who
received a smallpox vaccine during this time may have incorrectly
reported receiving this vaccine on the questionnaire.1 The agreement
for those who were born before 1960 (k = 0.51), who were likely
vaccinated during this period, is low, suggesting they may have only
received the smallpox vaccination as a child or as a military recruit.
Furthermore, anthrax vaccination may be more easily recalled due
to controversy surrounding the administration of the multi‑dose
vaccine.27,28
Individuals who self‑reported smallpox vaccination but did not
have electronic confirmation were slightly less likely to have deployed
in support of GWOT and more likely to be in the Navy/Coast
Guard and work in functional support roles, when compared with
participants whose electronic and self‑report indicated vaccination.
Differences among the service branches may be due to variations
in procedures for reporting as well as administering vaccinations.
For example, lower agreement (k = 0.41) among Navy personnel
could be explained by delays in shipboard vaccination reporting
to the electronic system. Those with no deployment experience
in support of GWOT had lower agreement (k = 0.35) between
electronic and self‑reported vaccination status than the study
population. The smallpox vaccination is often administered in
conjunction with deployment, therefore those with recent deployment experience may recall their vaccination more easily because
they were able to associate it with their deployment. The lower
agreement among those not deployed in support of GWOT
may also be associated with age, since those deploying tend to
be younger. Moreover, the older group of nondeployers may
be recalling childhood vaccinations. The lower kappa statistic
among those not recently deployed, however, may also be attributed
www.landesbioscience.com

to lower prevalence of vaccination among this group since the kappa
statistic is dependent on the true prevalence of the examined variable
and tends toward zero when the true prevalence is very high or
low.29
While there were some significant differences in SF‑36V scores
between the concordance groups, there was no apparent trend
of reporting lower health measures. For social functioning and limitations caused by physical and mental problems, participants whose
self‑reported vaccination was not confirmed by electronic record
had lower adjusted mean scores, indicating slightly worse health
than those with concordant electronic and self‑reported vaccination
status. Those whose vaccination status agreed had significantly
higher physical functioning and general health adjusted mean scores
compared with the other concordance groups; those who reported
no vaccination but had a vaccination recorded in the electronic
records had a lower adjusted mean score for vitality compared with
all other groups. In a previous study among Millennium Cohort
participants, lower adjusted mean scores were found for all eight
SF‑36V scales among participants whose self‑reported anthrax
vaccination was not confirmed in the electronic records.11 This indication that some Millennium Cohort participants may report more
exposures as well as more health challenges was not as strongly supported
among discordance of reporting in the current study. However,
consistent with the previous vaccination study using Millennium
Cohort data, there were no significant differences in the
hospitalization experience between the smallpox vaccination groups
for the one year prior to survey completion.11
In the unfortunate event of a smallpox bioterrorist attack,
vaccine doses will be limited in supply, and time will be of critical
importance. Results of the current study suggest that 96% of those
who report not being vaccinated are confirmed unvaccinated in the
electronic database whereas 61% of those who self‑report vaccination
have electronic documentation of vaccination. While the smallpox
vaccination scar may be used to identify those who are vaccinated,
self‑reported vaccination status could also be used as a screening
tool to correctly identify those who have not been vaccinated.
Though some individuals may be revaccinated, the biggest risk of
misclassification and possible smallpox infection and transmission,
would be among individuals who self‑report being vaccinated
who have not truly received the vaccination. Resources could then
be better used to verify vaccination within medical records
or immunization data for this smaller group who self‑report receiving
the smallpox vaccination.
This study has several limitations. The study population consists
of a sample of Millennium Cohort participants and may not be
representative of the military population in general. However,
investigation of potential biases in the Millennium Cohort have
found a well‑representative military cohort who report reliable data
and who are not influenced to participate by poor health prior
to enrollment (Wells T, Naval Health Research Center, unpublished
manuscript).11,17,30‑35 Since a considerable percentage of military
service members received a smallpox vaccine starting in early 2003,
the dependence of the kappa statistic on the true prevalence of the
examined variable should not substantially affect the findings of this
study.29 The Millennium Cohort questionnaire does not explain
the administration process or typical skin reaction of the smallpox
vaccination. Without this information participants may find it
difficult to identify and report which vaccinations they have received.
Finally, the amount of incomplete documentation for smallpox
vaccination in the database remains unknown. While medical

Human Vaccines

249

Validation of Smallpox Vaccination and Measures of Health

records may have provided additional vaccination history, recent
research indicates a strong level of agreement (sensitivity = 93.8%,
specificity = 87.0%) between electronically maintained anthrax
vaccination records and anthrax vaccination data abstracted from
medical charts.36
Despite these limitations, there are many strengths to these
analyses. This study is the first large, population‑based study
of concordance between self‑report and objective smallpox vaccination
data. The unique opportunity to compare these data sources allows
insight into the reliability of other self‑reported measures on the
questionnaire and is important for future longitudinal assessment
when only self‑reported data are available. The large sample size
of this study allows for robust comparison of self‑report and objective
measures of smallpox vaccination. Further, this study was able to link
both objective and subjective measures of morbidity to self‑reported
and electronic vaccination status.
Vaccination against biological agents is an important component
of protection against bioterrorist attacks. During a time of crisis
or bioterrorist attack, knowing the limitations and strengths of
self‑reported vaccination data is vital to the success and containment
of emergency response efforts. Further, understanding the limitations
and strengths of self‑reported data in research allows for appropriate
interpretation of results. Therefore, it is important to establish
the reliability of self‑reported data, such as smallpox vaccination,
and to investigate differences in reporting based on health metrics.
This analysis found substantial agreement between objectively
maintained records and self‑reported smallpox vaccination.
Further, no substantial differences in health metrics were found
among the small number of those whose recall of vaccination did
not concur with their objective records. Overall results support
that self‑report of smallpox vaccination is a reliable measure of this
important exposure.
Acknowledgements

We are indebted to the Millennium Cohort Study participants,
without whom these analyses would not be possible. We thank Scott
L. Seggerman and Greg D. Boyd from the Management Information
Division, Defense Manpower Data Center, Seaside, California.
Additionally, we thank Lacy Farnell; Gia Gumbs, MPH; Isabel
Jacobson, MPH; Travis Leleu; Robert Reed, MS; Katherine Snell;
Steven Spiegel; Damika Webb; Kari Welch, MA; and James Whitmer
from the Department of Defense Center for Deployment Health
Research, Naval Health Research Center, San Diego, California;
and Michelle Stoia, also from the Naval Health Research Center.
We also thank Karl E. Friedl, US Army Medical Research and Materiel
Command, Fort Detrick, Maryland. We appreciate the support
of the Henry M. Jackson Foundation for the Advancement
of Military Medicine, Rockville, MD.
Note

This work represents report 7–15, supported by the Department
of Defense, under work unit no. 60002. The views expressed in
this article are those of the authors and do not reflect the official
policy or position of the Department of the Navy, Department of
the Army, Department of the Air Force, Department of Defense,
Department of Veterans Affairs or the US Government. The
authors declare that they have no conflict of interest to disclose.
This research has been conducted in compliance with all applicable
federal regulations governing the protection of human subjects
in research (Protocol NHRC.2000.007).
In addition to the authors, the Millennium Cohort Study Team
includes Paul J. Amoroso, M.D., M.P.H. (Army Research Institute
250

of Environmental Medicine; Natick, Maryland); Edward J. Boyko,
M.D., M.P.H. (Seattle Epidemiologic Research and Information
Center; Veterans Affairs Puget Sound Health Care System);
Gary D. Gackstetter, M.D., D.V.M., M.P.H. (Department of
Preventive Medicine and Biometrics; Uniformed Services University
of the Health Sciences; Bethesda, Maryland); Gregory C. Gray,
M.D., M.P.H. (College of Public Health; University of Iowa;
Iowa City, Iowa); Tomoko I. Hooper, M.D., M.P.H. (Department
of Preventive Medicine and Biometrics; Uniformed Services
University of the Health Sciences; Bethesda, Maryland); James
R. Riddle, D.V.M., M.P.H. (Air Force Research Laboratory;
Wright-Patterson Air Force Base; Ohio).
References
1. Rotz LD, Dotson DA, Damon IK, Becher JA. Vaccinia (smallpox) vaccine: Recommendations
of the advisory committee on immunization practices (ACIP), 2001. MMWR Recomm Rep
2001; 50:1‑25, (quiz CE1‑7).
2. World Health Organization. The global eradication of smallpox: Final report of the global
commision for the certification of smallpox eradication. Geneva: 1980.
3. Centers for Disease Control and Prevention. Protecting americans: Smallpox vaccination
program. 2002, (Available at: http://www.bt.cdc.gov/agent/smallpox/vaccination/vaccination‑program‑statement.asp. Accessed January 5, 2006).
4. Alibek K. Biohazard. New York, NY: Random House Inc, 1999.
5. Poland GA, Grabenstein JD, Neff JM. The US smallpox vaccination program: A review
of a large modern era smallpox vaccination implementation program. Vaccine 2005;
23:2078‑81.
6. Andrews RM. Assessment of vaccine coverage following the introduction of a publicly
funded pneumococcal vaccine program for the elderly in Victoria, Australia. Vaccine 2005;
23:2756‑61.
7. Mac Donald R, Baken L, Nelson A, Nichol KL. Validation of self‑report of influenza and
pneumococcal vaccination status in elderly outpatients. Am J Prev Med 1999; 16:173‑7.
8. Mahan CM, Kang HK, Dalager NA, Heller JM. Anthrax vaccination and self‑reported
symptoms, functional status, and medical conditions in the national health survey of Gulf
War era veterans and their families. Ann Epidemiol 2004; 14:81‑8.
9. Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA, Jaen CR. How valid
are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care 1998;
36:851‑67.
10. Zimmerman RK, Raymund M, Janosky JE, Nowalk MP, Fine MJ. Sensitivity and specificity of patient self‑report of influenza and pneumococcal polysaccharide vaccinations among
elderly outpatients in diverse patient care strata. Vaccine 2003; 21:1486‑91.
11. Smith B, Leard CA, Smith TC, Reed RJ, Ryan MA. Anthrax vaccination in the Millennium
Cohort: Validation and measures of health. Am J Prev Med 2007; 32:347‑53.
12. Nichol KL, Korn JE, Baum P. Estimation of outpatient risk characteristics and influenza
vaccination status: Validation of a self‑administered questionnaire. Am J Prev Med 1991;
7:199‑203.
13. Shenson D, Dimartino D, Bolen J, Campbell M, Lu PJ, Singleton JA. Validation of self‑reported pneumococcal vaccination in behavioral risk factor surveillance surveys: Experience
from the sickness prevention achieved through regional collaboration (SPARC) program.
Vaccine 2005; 23:1015‑20.
14. Huerta M, Balicer RD, Mimouni D, Goldberg A, Furman MM, Klement E, Hourvitz
A, Grotto I. Validity of self‑assessment of skin reaction after smallpox vaccination. Public
Health Rep 2006; 121:45‑50.
15. Olmsted SS, Grabenstein JD, Jain AK, Comerford W, Giambo P, Johnson P, Mopsik J,
Zimmerman SR, Lurie N. Use of an electronic monitoring system for self‑reporting smallpox vaccine reactions. Biosecur Bioterror 2005; 3:198‑206.
16. Gray GC, Chesbrough KB, Ryan MA, Amoroso P, Boyko EJ, Gackstetter GD, Hooper TI,
Riddle JR. The Millennium Cohort Study: A 21‑year prospective cohort study of 140,000
military personnel. Mil Med 2002; 167:483‑8.
17. Ryan MA, Smith TC, Smith B, Amoroso P, Boyko EJ, Gray GC, Gackstetter GD, Riddle
JR, Wells TS, Gumbs G, Corbeil TE, Hooper TI. Millennium Cohort: Enrollment begins
a 21‑year contribution to understanding the impact of military service. J Clin Epidemiol
2007; 60:181‑91.
18. Kazis LE, Lee A, Spiro IIIrd A, Rogers W, Ren XS, Miller DR, Selim A, Hamed A, Haffer
SC. Measurement comparisons of the medical outcomes study and veterans SF‑36 health
survey. Health Care Financ Rev 2004; 25:43‑58.
19. Kazis LE, Miller DR, Clark JA, Skinner KM, Lee A, Ren XS, Spiro IIIrd A, Rogers WH,
Ware Jr JE. Improving the response choices on the veterans SF‑36 health survey role
functioning scales: Results from the veterans health study. J Ambul Care Manage 2004;
27:263‑80.
20. Kazis LE, Miller DR, Skinner KM, Lee A, Ren XS, Clark JA, Rogers WH, Spiro IIIrd A,
Selim A, Linzer M, Payne SM, Mansell D, Fincke RG. Patient‑reported measures of health:
The veterans health study. J Ambul Care Manage 2004; 27:70‑83.
21. Ware Jr JE, Sherbourne CD. The MOS 36‑item short‑form health survey (SF‑36). I.
Conceptual framework and item selection. Med Care 1992; 30:473‑83.

Human Vaccines

2007; Vol. 3 Issue 6

Validation of Smallpox Vaccination and Measures of Health
22. Ware Jr JE, Kosinski M. SF‑36 health survey: Manual and interpretation guide. Lincoln, RI:
2000.
23. Cohen JA. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;
20:37‑46.
24. Landis JR, Koch GG. The measurement of observer agreement for categorical data.
Biometrics 1977; 33:159‑74.
25. Use of anthrax vaccine in the United States. MMWR Recomm Rep 2000; 49:1‑20.
26. Cohen J. Bioterrorism. Smallpox vaccinations: How much protection remains? Science
2001; 294:985.
27. General accounting office. Medical readiness: DoD continues to face challenges in implementing its anthrax vaccine immunization program. Washington, DC: 2000.
28. In: Joellenbeck LM, Zwanziger LL, Durch JS, Strom BL, eds. Institute of Medicine:
Committee to assess the safety and efficacy of the anthrax vaccine MF‑UA. The anthrax
vaccine: Is it safe? Does it work? Washington, DC: National Academies Press, 2002.
29. Thompson WD, Walter SD. A reappraisal of the kappa coefficient. J Clin Epidemiol 1988;
41:949‑58.
30. Riddle JR, Smith TC, Smith B, Corbeil TE, Engel CC, Wells TS, Hoge CW, Adkins J,
Zamorski M, Blazer D. Millennium Cohort: The 2001‑2003 baseline prevalence of mental
disorders in the U.S. military. J Clin Epidemiol 2007; 60:192‑201.
31. Smith TC, Smith B, Jacobson IG, Corbeil TE, Ryan MA. Reliability of standard health
assessment instruments in a large, population‑based cohort study. Ann Epidemiol 2007;
17:525-32.
32. Chretien JP, Chu LK, Smith TC, Smith B, Ryan MA. Demographic and occupational
predictors of early response to a mailed invitation to enroll in a longitudinal health study.
BMC Med Res Methodol 2007; 7:6.
33. Smith TC, Jacobson IG, Smith B, Hooper TI, Ryan MA. The occupational role of women
in military service: Validation of occupation and prevalence of exposures in the Millennium
Cohort Study. Int J Environ Health Res 2007; 17:271-84.
34. Smith B, Wingard DL, Ryan MAK, Macera CA, Patterson TL, Slymen DJ. US military
deployment during 2001–2006: Comparison of subjective and objective data sources in a
large prospective health study. Ann Epidemiol 2007; In press.
35. Smith B, Smith TC, Gray GC, Ryan MAK. When epidemiology meets the Internet: Webbased surveys in the Millennium Cohort Study. Am J Epidemiol 2007; In press.
36. Payne DC, Rose Jr CE, Aranas A, Zhang Y, Tolentino H, Weston E, McNeil MM, Ruscio
B. Assessment of anthrax vaccination data in the Defense Medical Surveillance System,
1998‑2004. Pharmacoepidemiol Drug Saf 2007.

www.landesbioscience.com

Human Vaccines

251