MilCohort Prior health care utilization as a determinant of enrollment in Millennium Cohort Feb08 Eur J Epi

Eur J Epidemiol (2008) 23:79–87
DOI 10.1007/s10654-007-9216-0

METHODS

Prior health care utilization as a potential determinant
of enrollment in a 21-year prospective study, the Millennium
Cohort Study
Timothy S. Wells Æ Isabel G. Jacobson Æ Tyler C. Smith Æ Christina N. Spooner Æ
Besa Smith Æ Robert J. Reed Æ Paul J. Amoroso Æ Margaret A. K. Ryan Æ
For the Millennium Cohort Study Team

Received: 16 March 2007 / Accepted: 20 December 2007 / Published online: 10 January 2008
Ó Springer Science+Business Media B.V. 2008

Abstract Results obtained from self-reported health data
may be biased if those being surveyed respond differently
based on health status. This study was conducted to

In addition to the authors, the Millennium Cohort Study Team
includes Edward J. Boyko, MD, MPH (Seattle Epidemiologic
Research and Information Center, Department of Veterans Affairs
Puget Sound Health Care System, Seattle, WA); Gary D. Gackstetter,
PhD, DVM, MPH and Tomoko I Hooper, MD MPH (Department of
Preventive Medicine and Biometrics, Uniformed Services University
of the Health Sciences, Bethesda, MD); Gary D. Gackstetter, PhD,
DVM, MPH (Analytic Services, Inc. (ANSER), Arlington, VA);
Gregory C. Gray, MD, MPH (College of Public Health, University of
Iowa, Iowa City, IA); and James R. Riddle, DVM, MPH (Air Force
Research Laboratory, Wright-Patterson Air Force Base, OH).
This represents report 07-07, supported by the Department of
Defense, under work unit no. 60002. The views expressed in this
article are those of the authors and do not reflect the official policy or
position of the Department of the Navy, Department of the Army,
Department of the Air Force, Department of Defense, Department of
Veterans Affairs, or the US Government. This research has been
conducted in compliance with all applicable federal regulations
governing the protection of human subjects in research (Protocol
NHRC.2000.007).
T. S. Wells (&)
Biomechanics Branch, Air Force Research Laboratory, WrightPatterson Air Force Base, 2800 Q Street, Building 824, Room
206, Dayton, OH, USA
e-mail: [email protected]
I. G. Jacobson
T. C. Smith
C. N. Spooner
B. Smith

R. J. Reed
M. A. K. Ryan
Department of Defense Center for Deployment Health Research,
Naval Health Research Center, San Diego, CA, USA
P. J. Amoroso
Madigan Army Medical Center, Tacoma, WA, USA
For the Millennium Cohort Study Team
San Diego, CA, USA

investigate if health, as measured by health care use preceding invitation, influenced response to invitation to a 21year prospective study, the Millennium Cohort Study.
Inpatient and outpatient diagnoses were identified among
more than 68,000 people during a one-year period prior to
invitation to enroll. Multivariable logistic regression
defined how diagnoses were associated with response.
Days spent hospitalized or in outpatient care were also
compared between responders and nonresponders. Adjusted odds of response to the questionnaire were similar over
a diverse range of inpatient and outpatient diagnostic categories during the year prior to enrollment. The number of
days hospitalized or accessing outpatient care was very
similar between responders and nonresponders. Study
findings demonstrate that, although there are some small
differences between responders and nonresponders, prior
health care use did not affect response to the Millennium
Cohort Study, and it is unlikely that future study findings
will be biased by differential response due to health status
prior to enrollment invitation.
Keywords Cohort studies
Military medicine

Military personnel
Response bias
Veterans
Abbreviations
CI
Confidence interval
ICD-9International Classification of Diseases, Ninth
CM
Revision, Clinical Modification
OR
Odds ratio

Introduction
The decision to participate in epidemiologic health studies
is multifaceted, and information on nonresponse is often

123

80

lacking for a variety of reasons. It has been well documented that certain groups may be less likely to respond to
surveys, including men, young adults, those of lower
socioeconomic status, and unmarried individuals [1–4].
Basic survey research asserts that the prime motivation to
respond is based on the perceived salience of the study to
the subject. Heberlein and Baumgartner conducted a statistical analysis of over 90 methodological articles
published on response rate to mailed questionnaires, and
concluded that the number of subject contacts and the
importance of the questionnaire were the largest determinants to response rate [5].
In the conduct of an epidemiologic health study that
utilizes self-reported data, the investigators should be
cognizant that response bias may occur if invitees choose
to participate based on actual or perceived differences in
health status. To evaluate the presence or absence of
response bias contingent upon health status, the investigators must have access to health information on
responders and nonresponders. Since health care usage data
are largely unavailable for nonresponders, it may be very
difficult for US investigators to assess the health status of
these individuals. In fact, most studies of nonresponder
health have been conducted in Europe where universal
health coverage is more common [1–4, 6–9]. Several of
these studies found that individuals who were more likely
to respond to health surveys were those who frequently
experienced pain and/or utilized health care [6, 8, 9],
among the ‘‘worried well’’ [1], reported poorer subjective
health and less-healthy lifestyle habits [3], and were not
receiving a disability benefit [2, 4].
Results from these European studies prompted investigation into whether a response bias due to health
differences prior to enrollment exists among a US military
population in the Millennium Cohort Study. Current literature suggests that a response bias due to differential health
status between responders and nonresponders would likely
cause more individuals with poor health to join the Cohort
[6, 8, 9]. Should this participation bias occur and remain
unknown, it may have significant inferential limitations or
even lead to misleading conclusions. For nonresponse bias
to impact study findings, the respondents and nonrespondents must differ on reported data and the response rate
must be sufficiently low so that this difference can have an
appreciable effect [10]. For example, with a low response
rate, prevalence estimates derived from cross-sectional
surveys may be biased if these two conditions exist to an
appreciable degree. In contrast, when conducting a longitudinal study, the effect of response bias may be less
intrusive as the outcome measure is often assessed at follow-up and dependent upon baseline health status. Because
the Millennium Cohort Study is a longitudinal prospective
study, nonresponse bias is less likely to affect results in

123

T. S. Wells et al.

comparison to studies of cross-sectional design. Yet, it
remains prudent for researchers to investigate where possible, response biases that may result from self-selection
and over or under-representation of healthy or ill participants in a study population. In a longitudinal health study
such as this, military personnel of poorer health are less
likely to deploy and be exposed to deployment related
exposures. This would likely bias prevalence estimates and
might impact longitudinal studies, limiting the utility of the
Cohort.
Because all inpatient and outpatient encounters for
medical care under the US Military Health System are
electronically recorded for active-duty service members,
this study represents a unique opportunity to study response
bias based on the health care use of both responders and
nonresponders. Furthermore, these results may be generalizable to the other populations. For example, previous
reports have assessed the similarities in the distribution of
job titles among members of the US civilian workforce and
US military. Sulsky has compared the distribution of job
titles among members of the US Army who were serving
on active duty in 2001 to information published in the 2000
US Bureau of Labor Statistics, Occupational Employment
Statistics Survey. With the exception of soldiers working in
military-specific jobs, there was a high degree of correlation in the types of job titles. While the percent of civilian
and military workforce in transportation, and construction
were similar, the percent of military working in other jobs
was over- or under-represented compared to their civilian
counterparts [11]. In summary, it is important to understand possible biases associated with the Millennium
Cohort Study, the largest prospective study to ever be
conducted with a military population. Additionally, this
study may provide one of the few opportunities to accurately assess the health of invitees in the year prior to study
invitation in a large population-based Cohort of US military personnel, comparable in many ways to civilian
populations.

Materials and methods
Population and data sources
The invited Millennium Cohort Study participants were
randomly selected from all US military personnel serving
in October 2000, oversampling those who had been previously deployed, Reserve and National Guard members,
and women, to ensure sufficient power to detect differences
in smaller subgroups of the population. The probabilitybased sample, representing *11.3% of the 2.2 million men
and women in service as of October 1, 2000, was provided
by the Defense Manpower Data Center in California.

Prior health of the Millennium Cohort

World Wide Web and US Postal Service-based enrollment
began in July 2001, and a modified Dillman approach was
utilized to contact the invited service members. The
enrollment cycle ended on June 30, 2003, with 77,047
consenting participants. The methodology of the Millennium Cohort Study has been described elsewhere in detail
[12]. For the purposes of this analysis, only those invited
personnel on active duty were considered (n = 140,842),
since comparable health care utilization records for
Reserve and National Guard personnel may not exist. The
population for this study was further restricted by excluding those active-duty service members who were deployed
to a combat area at any time during the year prior to
enrollment or during the enrollment cycle itself, due to the
likelihood of differential access to care. This research was
conducted in compliance with all applicable federal regulations governing the protection of human subjects in
research (Protocol NHRC.2000.007).
Electronic personnel data as of sample construction in
October 2000 included gender, birth date, highest education level, marital status, race/ethnicity, service branch
(Army, Navy, Coast Guard, Air Force, and Marine Corps),
primary and secondary military occupations, and a personal
identifier. Additionally, missing electronic data on age,
education, marital status, race/ethnicity, and occupation
were supplemented with available electronic personnel
data from July 2000 to January 2001. This process of
backfilling missing information reduced the percentage of
individuals missing data for at least one important demographic characteristic from 2.8% to 1.1% of the invited
personnel, resulting in an analysis population of 68,103.

Health care data
To assess the effect of prior health care use on enrollment
into the Millennium Cohort, the frequency and types of
health care encounters were captured using Department of
Defense electronic hospitalization and outpatient records
during the period from July 1, 2000 to June 30, 2001.
Through an existing data use agreement, worldwide electronic hospitalization and outpatient records for all US
service members were accessed through a secure portal.
Through this service, hospitalization and outpatient records
for all US service members were downloaded to a secure
server and then matched to study participants by personal
identifier. Because healthcare is provided at no cost to all
US service members, we expect to non-differentially capture all hospitalizations and outpatient visits for both
responders and nonresponders represented in this study.
Hospitalization data included admission date and up to
eight discharge diagnoses, while outpatient data included
care date with up to four diagnoses. Diagnoses were coded

81

according to the International Classification of Diseases,
Ninth Revision, Clinical Modification (ICD-9-CM) [13].
Odds of response to the Millennium Cohort questionnaire
among those hospitalized for ‘‘any cause’’ were examined,
excluding diagnoses for complications of pregnancy,
childbirth, and puerperium. In addition, each of the 15
broad ICD-9-CM categories, and the supplementary categories for factors influencing health status and contact
with health services, were investigated separately to calculate the odds of response to the questionnaire among
those with a diagnosis in each category. Analysis of
individual diagnoses within the broad diagnostic categories of hospitalizations significantly associated with
response focused on the five most frequent three-digit
diagnostic codes to examine their association with
response. Additionally, specific ICD-9-CM diagnoses that
have previously been indicative of chronic multisymptom
illness were explored [14–25]. To further evaluate health
care usage differences between responders and nonresponders, the mean number of days spent hospitalized or
in outpatient care during the year prior to enrollment was
also compared.

Statistical analysis
Descriptive analyses of demographic and military characteristics, by both hospitalization and survey response status,
were completed. Univariate analyses, including v2 tests,
were employed to assess the significance of unadjusted
associations. An exploratory analysis was conducted to
examine regression diagnostics, significant associations,
and possible confounding, while simultaneously adjusting
for other variables. Multivariable logistic regression was
used to compare the adjusted odds of response to the
Millennium Cohort questionnaire among those with and
without hospitalization experience for any cause during the
year prior to enrollment. Individual multivariable logistic
regression models [26] were constructed for each diagnostic category, as previously defined. All models were
adjusted for age, gender, race/ethnicity, education, marital
status, military pay grade, military branch of service, and
military occupation.
Lastly, the mean number of days that responders and
nonresponders utilized inpatient, outpatient, and both types
of care was calculated and compared by response status
using analysis of variance. Of the 3,833 individuals who
appeared in the hospitalization database, 207 did not have a
computable length of stay, so the mean length of stay for
all hospitalizations (mean = 3.499 days) was assigned.
The total number of days lost to care over the year was
considered to be total days hospitalized plus one half-day
lost to care for each outpatient visit.

123

82

All analyses were completed using SAS software (version 9.1, SAS Institute, Inc., Cary, North Carolina), and
adjusted odds ratios (ORs), 95% confidence intervals (CIs),
and adjusted means were calculated for personnel with
complete covariate data [27].

Results
Descriptive analyses are shown in Table 1. Those more
likely to be hospitalized were women, those younger than
24 or older than 45 years of age, those with a high school
diploma or less, enlisted personnel, those not married,
members of the Army, and participants in various occupations, most notably health care workers (p \ 0.05).
Several demographic factors were also associated with
survey response (Table 2). There was no association,
however, between any-cause hospitalization and response
(adjusted OR = 0.93, 95% CI: 0.85, 1.02) after adjusting
for gender, age, race/ethnicity, educational level, military
rank, marital status, service branch, and occupation.
Millennium Cohort responders and nonresponders were
also evaluated for the time spent accessing inpatient or
outpatient care in the year prior to enrollment. The mean
and median number of days spent in aggregated inpatient
or outpatient care differed by less than a half-day and was
statistically significant at the a = 0.05 level (Table 3).
In separate multivariable logistic regression analyses,
adjusted odds of response were calculated for 16 broad
categories of ICD-9-CM coded diagnoses. There was no
association between response and hospitalizations, except
that responders were at significantly lower adjusted odds
for mental disorder hospitalizations (Fig. 1). When outpatient diagnoses were examined, responders were less likely
to have outpatient visits for mental disorders (OR = 0.88,
95% CI: 0.83, 0.93). In addition, responders were significantly more likely to have outpatient visits for neoplasms;
diseases of the nervous, circulatory, musculoskeletal, and
respiratory systems; diseases of the skin and subcutaneous
tissues; endocrine, nutritional, and metabolic diseases; and
for supplementary (V) codes, after adjusting for all variables in the models, although these adjusted ORs were all
less than 1.2 (Fig. 2).
Those ICD-9-CM categories with statistically significant
results were further evaluated to identify the top-five most
frequent diagnoses. The five most common inpatient
mental disorder diagnoses included nondependent abuse of
drugs, adjustment reactions, affective psychoses, alcohol
dependence syndrome, and personality disorders; those
hospitalized for adjustment reactions and personality disorders were significantly less likely to respond (Table 4).
Because diseases such as asthma, fibromyalgia, and chronic
fatigue syndrome were reported to be more prevalent in

123

T. S. Wells et al.

1990–1991 Gulf War veterans than nondeployed veterans
of the same era, we sought to determine if there was a
response bias based on these medical conditions. After
adjusting for gender, age, education, marital status, race/
ethnicity, pay grade, branch of service, and occupation,
there was no association between response status and
asthma, fibromyalgia, or chronic fatigue syndrome as
measured by either hospitalizations or outpatient visits
(data not shown).

Discussion
In this study, we found no substantial health differences
between Millennium Cohort responders and nonresponders
when measured by health care use in the 12 months preceding study invitation. With the exception of mental
disorders, there were no differences in response based on
prior hospitalization experiences between responders and
nonresponders. Responders were slightly more likely to
have utilized outpatient health care across eight broad ICD9-CM categories, although the small ORs observed were
close to unity, implying little to no clinically meaningful
differences. Additionally, findings demonstrated that
responders and nonresponders were very similar in terms of
the number of days spent utilizing inpatient and outpatient
health care services.
Responders and nonresponders appeared to be of comparable health based on inpatient data. This finding likely
provides the best assessment that responders and nonresponders did not have significant health differences in the
year prior to invitation for enrollment. Becoming hospitalized generally involves an objective decision by a
medical provider indicating the patient’s condition warrants hospitalization. In contrast, the decision to seek
outpatient care may be a personal decision to seek consultation, preventive medicine, or medical treatment
services.
In further analyses of outpatient visits (data not shown),
we observed that responders were more likely to have an
outpatient diagnosis for unspecified neoplasms, disorders
of lipoid metabolism, diseases of capillaries, contact dermatitis and other eczema, other and unspecified disorders
of joints, and acute pharyngitis, and allergic rhinitis, all of
which appear to indicate that responders may have a higher
disease burden than nonresponders. However, responders
were also more likely to have sought medical consultation
without complaint or sickness, to have completed followup examinations, or had a visit to correct a disorder of
vision refraction and accommodation, indicating that
responders also appear to utilize health care for reasons
other than disease or injury treatment. Additionally, individual decisions to seek outpatient health care may be

Prior health of the Millennium Cohort
Table 1 Characteristics
associated with hospitalization
in the year prior to invitation
among active-duty members
invited to participate in the
Millennium Cohort in 2001a

Characteristicsb

83

Total
(n = 68,103)
n (%)

Hospitalized
(n = 2,335)
n (%)

Not hospitalized
(n = 65,768)
n (%)

Male

50,447 (74.1)

1,543 (66.1)

48,904 (74.4)

Female

17,656 (25.9)

792 (33.9)

16,864 (25.6)

17–24

19,054 (28.0)

714 (30.6)

18,340 (27.9)

25–34

17,709 (26.0)

563 (24.1)

17,146 (26.1)

35–44
C45

18,101 (26.6)
13,239 (19.4)

543 (23.3)
515 (22.1)

17,558 (26.7)
12,724 (19.4)

White, non-Hispanic

41,471 (60.9)

1,404 (60.1)

40,067 (60.9)

Black, non-Hispanic

15,185 (22.3)

563 (24.1)

14,622 (22.2)

Other

11,447 (16.8)

368 (15.8)

11,079 (16.9)

High school diploma or less

40,433 (59.4)

1,560 (66.8)

38,873 (59.1)

Some college

18,587 (27.3)

530 (22.7)

18,057 (27.5)

Bachelor’s degree

4,976 (7.3)

125 (5.4)

4,851 (7.4)

Graduate school

4,107 (6.0)

120 (5.1)

3,987 (6.1)

Enlisted

60,828 (89.3)

2,160 (92.5)

58,668 (89.2)

Officer

7,275 (10.7)

175 (7.5)

7,100 (10.8)

25,019 (36.7)
39,826 (58.5)

888 (38.0)
1,313 (56.2)

24,131 (36.7)
38,513 (58.6)

3,258 (4.8)

134 (5.7)

3,124 (4.8)

Army

22,909 (33.6)

1,017 (43.6)

21,892 (33.3)

Air Force

20,483 (30.1)

576 (24.7)

19,907 (30.3)

Navy and Coast Guard

18,000 (26.4)

554 (23.7)

17,446 (26.5)

6,711 (9.9)

188 (8.1)

6,253 (9.9)

Gender

Age (years)

Race/ethnicity

Highest educational level

Rank

Marital status
Single
Married
Other
Service branch

Marine Corps
a

Selected based on the
following criteria: must have
served on continuous activeduty throughout health care
observation period and not
deployed at any time during the
health care observation period
or study enrollment period

b

Characteristics defined as of
October 2000 when the
potential participant pool was
identified. All v2 tests of
significance, except race/
ethnicity, were statistically
significant at P \ 0.05

Occupational codes
Combat specialists

12,522 (18.4)

442 (18.9)

12,080 (18.4)

Functional support and admin

14,567 (21.4)

509 (21.8)

14,058 (21.4)

Electrical/mechanical equip.
repair

11,442 (16.8)

373 (16.0)

11,069 (16.8)

Electronic equipment repair
Health care

6,383 (9.4)
6,228 (9.1)

179 (7.7)
267 (11.4)

6,204 (9.4)
5,961 (9.1)

Communications/intelligence

5,627 (8.3)

168 (7.2)

5,459 (8.3)

Service and supply

5,567 (8.2)

217 (9.3)

5,350 (8.1)

Craft workers

2,102 (3.1)

64 (2.7)

2,038 (3.1)

Other technical and allied

1,862 (2.7)

50 (2.1)

1,812 (2.8)

Students, prisoners, and other

1,803 (2.7)

66 (2.8)

1,737 (2.6)

influenced by accessibility, perceived condition severity,
concerns over social stigmatization associated with healthrelated personal habits, or illness conditions. Furthermore,
military members who receive flight and other specialty
pay jeopardize these earnings should they be diagnosed with
certain duty-limiting medical conditions. All of these possibilities make the use of outpatient care a more

ambiguous measure of health in comparison with inpatient
health care use.
Invitees with inpatient or outpatient visits for mental
disorders were significantly less likely to participate in the
study. Upon closer examination of these data, nondependent abuse of drugs and adjustment reactions were the two
most frequent three-digit ICD-9-CM diagnoses in both the

123

84
Table 2 Characteristics
associated with response to
invitation to enroll in the
Millennium Cohort among
active-duty members invited in
2001

T. S. Wells et al.
Characteristicsa

Non-responders
(n = 47,036)
n (%)

Responders
(n = 21,067)
n (%)

ORb 95% CIb

Hospitalization in year prior to invitation
Not hospitalized
Hospitalized

45,403 (96.5)
1,633 (3.5)

20,365 (96.7)
702 (3.3)

1.00
0.93 0.84, 1.03

Gender
Male

35,304 (75.1)

15,143 (71.9)

Female

11,732 (24.9)

5,924 (28.1)

1.43 1.37, 1.49

Age (years)
17–24

15,520 (33.0)

3,534 (16.8)

1.00

25–34

12,988 (27.6)

4,721 (22.4)

1.47 1.39, 1.55

35–44

11,067 (23.5)

7,034 (33.4)

2.45 2.31, 2.59

7,461 (15.9)

5,778 (27.4)

2.71 2.54, 2.89

White, non-Hispanic

27,734 (59.0)

13,737 (65.2)

Black, non-Hispanic

11,568 (24.6)

3,617 (17.2)

0.56 0.53, 0.58

7,734 (16.4)

3,713 (17.6)

0.81 0.77, 0.85

High school diploma or less

29,459 (62.6)

10,974 (52.1)

Some college

12,745 (27.1)

5,842 (27.7)

C45

1.00

Race/ethnicity

Other

1.00

Highest educational level
1.00
1.15 1.07, 1.23

Bachelor’s degree

2,842 (6.0)

2,134 (10.1)

1.16 1.07, 1.27

Graduate school

1,990 (4.2)

2,117 (10.1)

1.20 1.06, 1.35

43,328 (92.1)
3,708 (7.9)

17,500 (83.1)
3,567 (16.9)

1.00
1.45 1.32, 1.60

Rank
Enlisted
Officer
Marital status
Single

19,536 (41.5)

5,483 (26.0)

Married

25,428 (54.1)

14,398 (68.3)

1.28 1.23, 1.34

2,072 (4.4)

1,186 (5.6)

1.15 1.05, 1.25

Army

15,115 (32.1)

7,794 (37.0)

1.00

Air Force

14,060 (29.9)

6,423 (30.5)

0.60 0.56, 0.64

Navy and Coast Guard

12,770 (27.2)

5,230 (24.8)

0.64 0.61, 0.67

5,091 (10.8)

1,620 (7.7)

0.67 0.63, 0.72

Other
a

Selected based on the
following criteria: must have
served on continuous active
duty throughout health care
observation period and not
deployed at any time during the
health care observation period
or study enrollment period.
Characteristics defined as of
October 2000 when the
potential participant pool was
identified

b

ORs and associated 95% CIs
from multiple logistic
regression were adjusted for
gender, age, education, marital
status, race/ethnicity, pay grade,
branch of service and
occupation. CIs that exclude
1.00 were significant at the
P \ 0.05 level

Service branch

Marine Corps
Occupational codes
Combat specialists

8,915 (19.0)

3,607 (17.1)

1.00

Functional support and admin

9,887 (21.0)

4,680 (22.2)

1.21 1.14, 1.28

Electrical/mechanical equip.
repair

8,241 (17.5)

3,201 (15.2)

1.22 1.15, 1.30

Electronic equipment repair

4,193 (8.9)

2,190 (10.4)

1.41 1.31, 1.51

Health care

3,954 (8.4)

2,274 (10.8)

1.32 1.23, 1.41

Communications/intelligence

3,791 (8.1)

1,836 (8.7)

1.32 1.23, 1.42

Service and supply

3,947 (8.4)

1,620 (7.7)

1.09 1.02, 1.18

Craft workers

1,540 (3.3)

562 (2.7)

1.13 1.01, 1.26

Other technical and allied

1,250 (2.7)

612 (2.9)

1.30 1.17, 1.46

Students, prisoners, and other

1,318 (2.8)

485 (2.3)

1.22 1.08, 1.37

inpatient and outpatient setting. Responders were significantly less likely to have inpatient or outpatient diagnoses
of adjustment reactions, and significantly less likely to have

123

1.00

outpatient visits for nondependent abuse of drugs. Finding
less participation by those with mental disorders is not
widely reported. In an analysis of nonparticipants for the

Prior health of the Millennium Cohort

85

Table 3 Number of days spent in inpatient and outpatient care in the
year prior to invitation among active-duty Millennium Cohort survey
responders and nonresponders
Median, IQRb

P-valuec

Inpatient hospitalization
Responder

0.21

0.00, 0.00

Nonresponder
0.25
Outpatient encounters

0.00, 0.00

Responder

4.94

3.00, 5.50

Nonresponder

4.58

2.50, 5.00

Responder

5.15

3.00, 5.50

Nonresponder

4.83

3.00, 5.00

0.034

1.10

1.00

0.90

\0.001
0.80

Combined

0.70

\0.001

P-value significance was calculated by analysis of variance

s
on
er d
iti
rd loo ns
nd
iso e B atio s Co
l D tb ic se d
ta f pl ea ne
en s o m is efi
M ase Co m D - D
y
ise c te 11
D nan Sys s, I c
eg e ign iti
es
Pr stiv , S ras
as
e s a
ig m P g ise
D pto and nin D es s
m n iso em as se
Sy ctio Po yst ise isea
fe nd S D D es
In y a ary em m eas
t
e
r
n
s
ju ri ys st i
In itou y S l Sy us D es c
r
i
en to ta eo as ol
G ira kele tan ise tab
s p o s cu D e )
Re cul ub tem l, M des
us d S ys na co
M an y S itio (V
in or utr s s
Sk ulat , N ode ase
rc ne l C ise
Ci ocri nta D
d e em
En lem yst
pp S
Su ous s
v
er m
N plas
eo

Mean number of days was calculated for the year prior to Millennium Cohort Study enrollment, July 1, 2000, to June 30, 2001
b
Interquartile range indicates the difference between the third
quartile and the first quartile

N

a

c

1.20

Adjusted Odds Ratios

Meana

Data source

1.30

s

Canadian Study of Health and Aging, the authors found that
cognitively impaired subjects had higher refusal rates [28].
It is also possible that individuals diagnosed with nondependent abuse of drugs and other mental disorders feel a
certain degree of social stigmatization and prefer not to
share their health information with strangers. For example,
Hoge et al. documented that some military members are
likely to avoid seeking health care for mental disorders due
to perceived concerns over the social stigmatization and the
1.50
1.40

Adjusted Odds Ratios

1.30
1.20
1.10
1.00
0.90
0.80
0.70
0.60
0.50
s
er ic es
rd sit as ses
iso ara ise ea c
l D P D is li
ta nd em s D bo
en a st u ta
M ion Sy eo e
ct y an , M
fe or ut al
In at c n
s)
ul ub tio
rc S tri
de
Ci and Nu
co
V
in e,
s(
Sk crin
de
do ms Co
En as al ng es
pl nt ni as
eo e so se
N lem oi Di s
n
P
ns
pp d em tio
tio
Su y an yst lica
di
r S
on
ju e mp d
In stiv Co loo es d C
s e
e
ig cy e B ea in
D nan f th Dis Def es
eg o m 1- as
Pr ases ste , I1 ise
y s D s
e s
ise S gn
D ous , Si stem eas ase
v s y is e
er m S D is
N to ary m D
p
m rin ste tem
Sy tou Sy Sys
y
i
en or al
G irat elet
sp sk
Re culo
us

M

Fig. 1 Adjusted ORs and 95% CIs for odds of response to the
Millennium Cohort questionnaire, adjusted for gender, age, education,
marital status, race/ethnicity, pay grade, branch of service and
occupation for responders and nonresponders who were hospitalized
from July 1, 2000, to June 30, 2001, 1 year prior to the start of study
enrollment

Fig. 2 Adjusted ORs and 95% CIs for odds of response to the
Millennium Cohort questionnaire, adjusted for gender, age, education,
marital status, race/ethnicity, pay grade, branch of service and
occupation for responders and nonresponders who utilized outpatient
care from July 1, 2000, to June 30, 2001, 1 year prior to the start of
study enrollment

possible adverse impact mental disorder diagnoses may
have on careers [29, 30]. It is also plausible that those with
adjustment reactions felt overwhelmed at the time of invitation and felt that they were unable to add additional
responsibilities, such as committing to a 21-year follow-up,
thus making them less likely to respond.
These findings must be interpreted within the limitations
and strengths of the study. One potential limitation is that
this study was restricted to those Millennium Cohort
members whose health care data were available during the
entire observation period, which meant removing service
members in the Reserve or Guard and those who had
deployed during the observation period or during study
enrollment. This restriction was implemented to ensure
responders and nonresponders had equal opportunity to
access military health care. Because healthier service
members may be selected to deploy, this restriction may
have made this sample less representative of the entire
Millennium Cohort, or the US military in general. However, we have no reason to believe that this restriction
would have differentially selected responders or nonresponders with poorer health for these analyses.
In contrast to potential limitations, this study should be
considered robust by several measures. First, it was based
on 21,067 responders and 47,036 nonresponders, representing one of the largest studies ever of nonresponse bias to
enrollment invitation. Second, objective hospitalization and

123

86

T. S. Wells et al.

Table 4 Frequencies of the five most common inpatient diagnoses among statistically significant broad ICD-9-CM diagnostic categories for
Millennium Cohort Study responders and nonresponders with prior year hospitalizationsa
ICD-9-CM categoryb Diagnoses

Odds ratiod 95% CId

Frequency of diagnoses
Responders (n = 21,067)

Non-responders (n = 47,036)c

n

n

(%)

(%)

Mental disorders (codes 290–319)
305

Nondependent abuse of drugs

33

0.2

109

0.2

0.74

0.49, 1.10

309

Adjustment reaction

24

0.1

117

0.2

0.57

0.36, 0.90

296

Affective psychoses

30

0.1

80

0.2

0.96

0.62, 1.49

303

Alcohol dependence syndrome 14

0.1

55

0.1

0.59

0.32, 1.09

301

Personality disorders

0.0

52

0.1

0.26

0.09, 0.74

a

4

Hospitalizations were observed from July 1, 2000, to June 30, 2001, 1 year prior to the start of Millennium Cohort Study enrollment

b

ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification

c

The reference group for the multiple logistic regression models is nonresponders

d

ORs and associated 95% CIs from multiple logistic regression were adjusted for gender, age, education, marital status, race/ethnicity, pay
grade, branch of service and occupation. Measures in bold were statistically significant at the P \ 0.05 level

outpatient medical data were utilized to investigate diagnostic categories associated with enrollment as well as to
compute measures of days utilizing hospitalization and
outpatient care, allowing for a thorough analysis of this
subject. Finally, because military health care is equally
accessible, irrespective of gender, age, or pay grade, all
responders and nonresponders should have had equal access
to health care during the observation period, reducing the
potential for bias dependent on socioeconomic differences.
In conclusion, these analyses illustrated that responders
and nonresponders were comparable with regard to health
at the onset of the Millennium Cohort Study, as measured
by hospitalization and days spent accessing health care
services. No substantial or clinically relevant health-related
response bias was found among the participants, providing
reassurance that individuals in the Millennium Cohort
exhibit comparable health attributes that are representative
of the target population for this study.
Acknowledgments We are indebted to the Millennium Cohort
Study participants, without whom these analyses would not be possible. We thank Scott L. Seggerman from the Management
Information Division, Defense Manpower Data Center, Seaside,
California. Additionally, we thank Laura Chu, MPH; Lacy Farnell;
Gia Gumbs, MPH; Cynthia LeardMann, MPH; Travis Leleu; Steven
Spiegel; Damika Webb; Keri Welch, MA; and Jim Whitmer from the
Department of Defense Center for Deployment Health Research,
Naval Health Research Center, San Diego, California; and Michelle
Stoia, also from the Naval Health Research Center. We appreciate the
support of the Henry M. Jackson Foundation for the Advancement of
Military Medicine, Rockville, Maryland.

2.

3.

4.

5.

6.

7.

8.
9.

10.

11.

12.

References
13.
1. Boshuizen HC, Viet AL, Picavet HS, Botterweck A, van Loon
AJ. Non-response in a survey of cardiovascular risk factors in the

123

Dutch population: determinants and resulting biases. Public
Health. 2006;120(4):297–308.
Sogaard AJ, Selmer R, Bjertness E, Thelle D. The Oslo Health
Study: the impact of self-selection in a large, population-based
survey. Int J Equity Health. 2004;3(1):3.
Van Loon AJ, Tijhuis M, Picavet HS, Surtees PJ, Ormel J. Survey
non-response in the Netherlands: effects on prevalence estimates
and associations. Ann Epidemiol. 2003;13(2):105–10.
Korkeila K, Souminen S, Ahvenainen J, Ojanlatva A, Rautava P,
Helenius H, Koskenvuo M. Non-response and related factors in a
nation-wide health survey. Eur J Epidemiol. 2001;17(11):991–9.
Heberlein TA, Baumgartner R. Factors affecting response rates to
mailed questionnaires: a quantitative analysis of the published
literature. Am Sociol Rev. 1978;43(4):447–62.
Rupp I, Triemstra M, Boshuizen HC, Jacobi CE, Dinant HJ, van
den Bos GA. Selection bias due to non-response in a health
survey among patients with rheumatoid arthritis. Eur J Public
Health. 2002;12(2):131–5.
Kjoller M, Thoning H. Characteristics of non-response in the
Danish health interview surveys, 1987–1994. Eur J Public Health.
2005;15(5):528–35.
Etter JF, Perneger TV. Analysis of non-response bias in a mailed
health survey. J Clin Epidemiol. 1997;50(10):1123–8.
Freudenstein U, Arthur AJ, Matthews RJ, Jagger C. Community
surveys of late-life depression: who are the non-responders? Age
Ageing. 2001;30(6):517–21.
Bethel J, Greeen JL, Kalton G, Nord C. Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), Sampling. In: ECLS-B
Methodology Report for the 9-Month Data Collection,
2001–2002 (NCES 2005-147). U.S. Department of Education.
Vol. 2. Washington, DC: National Center for Education Statistics; 2005.
Sulsky SI. On occupational health and safety research in the US
Army: comparability with civilian employee cohorts. J Occup
Environ Med. 2003;45(3):220–1.
Ryan MA, Smith TC, Smith B, Amoroso P, Boyko EJ, Gray GC,
et al. Millennium Cohort: enrollment begins a 21-year contribution to understanding the impact of military service. J Clin
Epidemiol. 2007;60(2):181–91.
The International Classification of Diseases, 9th Revision. In:
Speirs L, editor. Clinical modification. 5th ed. Salt Lake City:
Medicode Publication 272; 1998.

Prior health of the Millennium Cohort
14. Baker DG, McQuarrie IG, Murray MG, Lund LM, Dashevsky
BA, Mendenhall CL. Diagnostic status and treatment recommendations for Persian Gulf War Veterans with multiple
nonspecific symptoms. Mil Med. 2001;166(11):972–81.
15. Bourdette DN, McCauley LA, Barkhuizen A, Johnston W, Wynn
M, Joos SK, et al. Symptom factor analysis, clinical findings, and
functional status in a population-based case control study of Gulf
War unexplained illness. J Occup Environ Med. 2001;43(12):
1026–40.
16. Cowan DN, Lange JL, Heller J, Kirkpatrick J, DeBakey S. A
case–control study of asthma among US Army Gulf War veterans
and modeled exposure to oil well fire smoke. Mil Med.
2002;167(9):777–82.
17. Eisen SA, Kang HK, Murphy FM, Blanchard MS, Reda DJ,
Henderson WG, et al. Gulf War veterans’ health: medical evaluation of a US cohort. Ann Intern Med. 2005;142(11):881–90.
18. Escalante A, Fischbach M. Musculoskeletal manifestations, pain,
and quality of life in Persian Gulf War veterans referred for
rheumatologic evaluation. J Rheumatol. 1998;25(11):2228–35.
19. Gray GC, Reed RJ, Kaiser KS, Smith TC, Gastanaga VM. Selfreported symptoms and medical conditions among 11,868 Gulf
War-era veterans: the Seabee Health Study. Am J Epidemiol.
2002;155(11):1033–44.
20. Kang HK, Natelson BH, Mahan CM, Lee KY, Murphy FM. Posttraumatic stress disorder and chronic fatigue syndrome-like illness among Gulf War veterans: a population-based survey of
30,000 veterans. Am J Epidemiol. 2003;157(2):141–8.
21. Karlinsky JB, Blanchard M, Alpern R, Eisen SA, Kang H,
Murphy FM, et al. Late prevalence of respiratory symptoms and
pulmonary function abnormalities in Gulf War I Veterans. Arch
Intern Med. 2004;164(22):2488–91.

87
22. Lange JL, Schwartz DA, Doebbeling BN, Heller JM, Thorne PS.
Exposures to the Kuwait oil fires and their association with
asthma and bronchitis among gulf war veterans. Environ Health
Perspect. 2002;110(11):1141–6.
23. McCauley LA, Joos SK, Barkhuizen A, Shuell T, Tyree WA,
Bourdette DN. Chronic fatigue in a population-based study of
Gulf War veterans. Arch Environ Health. 2002;57(4):340–8.
24. Smith TC, Gray GC, Knoke JD. Is systemic lupus erythematosus,
amyotrophic lateral sclerosis, or fibromyalgia associated with
Persian Gulf War service? An examination of Department of
Defense hospitalization data. Am J Epidemiol. 2000;151(11):
1053–9.
25. Young HA, Simmens SJ, Kang HK, Mahan CM, Levine PH.
Factor analysis of fatiguing syndrome in Gulf War era veterans:
implications for etiology and pathogenesis. J Occup Environ
Med. 2003;45(12):1268–73.
26. Hosmer DW, Lemeshow S. Applied logistic regression. New
York: John Wiley & Sons; 1989.
27. SAS Institute Inc., SAS/STATÒ 9.1 Users Guide. Cary, NC: SAS
Institute Inc.; 2004.
28. Helliwell B, Aylesworth R, McDowell I, Baumgarten M, Sykes
E. Correlates of nonparticipation in the Canadian Study of Health
and Aging. Int Psychogeriatr. 2001;13(Supp 1):49–56.
29. Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI,
Koffman RL. Combat duty in Iraq and Afghanistan, mental health
problems, and barriers to care. N Engl J Med. 2004;351(1):13–22.
30. Hoge CW, Lesikar SE, Guevara R, Lange J, Brundage JF, Engel
CC Jr, et al. Mental disorders among US Military personnel in the
1990s: association with high levels of health care utilization and
early military attrition. Am J Psychiatry. 2002;159:1576–83.

123