OMB Supporting_Statement_Part_B_27_JUN_2014_NHRC

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Panel	Dates Enrolled	Years of Service at Enrollment	Oversampled Groups	Roster Size (Date)	Number Contacted	Total Enrolled (% of contacted)
1	Jul 2001-Jun 2003	All durations (cross-section of military population)	Females, National Guard/ Reserves, and prior deployers*	256,400 (Oct 2000)	213,949	77,047 (36%)
2	Jun 2004-Feb 2006	1-2 years	Females and Marine Corps	150,000 (Oct 2003)	122,410	31,110 (25%)
3	Jun 2007-Dec 2008	1-3 years	Females and Marine Corps	200,000 (Oct 2006)	153,650	43,440 (28%)
4	June 2011-Ongoing	2-5 years	Females and Married	250,000 (Oct 2010)	Contact Ongoing	43,529 (17%)
Family	June 2011- Ongoing	N/A	Males	13,698**	Contact Ongoing	6,438** (47%)

**As of Nov 5, 2012; contact and enrollment still ongoing.

As shown in the above table, the responder universe for the Millennium Cohort Study consists of a probability-based sample of active-duty, Reserve, and National Guard members of the US military, identified through service rosters as of October 1, 2000 (Panel 1), October 1, 2003 (Panel 2), October, 1 2006 (Panel 3), and October 1, 2010 (Panel 4). Individuals invited to participate are not chosen based on location, whatsoever. While most invited individuals reside in the United States, service members can be stationed or deployed to almost any area of the world. Invitations are mailed to the current postal address of the service members, regardless of city, state, and country.

The responder universe for the Family Study consists of the spouses of married personnel with 2-5 years of service who enrolled in Panel 4 of the Millennium Cohort Study.

Follow-up Response Rates Total # of Responders per Cycle (% follow-up rate)
Panel (# in Panel)	2004-2006	2007-2008	2011-2012
Panel 1 (n=77,047)	55,021 (71%) Wave 2	54,790 (71%) Wave 3	Wave 4 is ongoing
Panel 2 (n=31,110)	Enrollment of Panel 2	17,151 (55%) Wave 2	Wave 3 is ongoing
Panel 3 (n=43,440)		Enrollment of Panel 3	Wave 2 is ongoing

**As of Nov 5, 2012; contact and enrollment still ongoing.

By July 1, 2003, 77,047 (36%) had returned a Panel 1 baseline questionnaire. Panel 1 targeted 256,400 service members of whom 213,949 had valid addresses allowing for study contact attempt. Among the 77,047 Panel 1 participants, 55,021 submitted the first follow-up survey (71%), 54,790 submitted the second follow-up survey (71%), and 51,199 (66%) submitted the third follow-up survey. Of those that did not submit their first follow-up survey, 8,259 submitted their second follow-up survey. Of the Panel 1 participants, 46,439 have completed the first three surveys.

Panel 2 targeted 150,000 of whom 122,410 were determined to have valid addresses allowing for study contact attempt. Of those, 31,110 (25%) returned a Panel 2 baseline questionnaire. Among Panel 2 participants, 17,151 have submitted a first follow-up survey (55%), and 14,893 (48%) submitted the second follow-up survey.

Panel 3 targeted 200,000 service members of whom 153,650 had documented valid addresses. Of those, 43,440 (28%) had returned a Panel 3 baseline questionnaire, and 21,672 (50%) completed their first follow-up survey.

Additionally for the 2011-2012 survey cycle, the Family Study is attempting to enroll 10,000 participants.

During the 2014-15 survey cycle, we expect to collect follow-up surveys on Panels 1-4 of the Millennium Cohort Study participants as well as the first panel of the Family Study.

To date, a total of 864 deaths have occurred within the Panel 1 responder group, 164 deaths have occurred within the Panel 2 responder group, and 143 deaths have occurred within Panel 3. As of November 5, 2012, there have been a total of 20 deaths within the Panel 4 responder group.

We estimate that approximately 46,747 of the participants from Panels 1, 2, 3, and 4 who respond to the 2014-2015 survey will not be military service members. Of the spouses who enrolled in the Family Study during 2011-2012, it is estimated that 3,576 will complete a follow-up survey between 2014 and 2015 and will not be military service members at that time. Therefore, for both Millennium Cohort and the Family Study it is estimated that a total of 50,323 participants who complete a survey between 2014 and 2015 will not be service members.

As of our first OMB approval in September 2003 and throughout the course of the study, proportions of military versus public, participants will shift in favor of members of the public.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The Millennium Cohort consists of service members randomly selected from a large, representative military sample obtained from the Defense Manpower Database Center. A probability-based random sampling process was employed with oversampling for certain sub-groups to ensure enough statistical power to address small subgroups of the population reasonably well in a population-based setting.

In Panel 1 military personnel who had served in Southwest Asia, Bosnia, and Kosovo after 1997 were over-sampled. Additionally, Reserve, National Guard, and female service personnel were over-sampled to assure sufficient statistical power to investigate hypotheses in these smaller subgroups of the military population. In Panels 2 and 3, military personnel who were Marines or females were oversampled to assure sufficient statistical power to investigate hypotheses in these smaller subgroups. In Panel 4, women and married service members were oversampled to support the enrollment of the concurrent Family Study. The 36,599 personnel who became members of the public during the 2011-12 effort (after enrolling into the Millennium Cohort Study) are expected to follow a random attrition process from military service and are expected to be similar in characteristics to the responders in general. The Millennium Cohort Study and Family Study participants are asked to complete a questionnaire every 3 years allowing information to be acquired without burdening participants with yearly questionnaires.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Much effort is focused into response and retention rates. Response rates for both the Millennium Cohort Study and Family Study are maximized principally through employment of modified Dillman Mail and Electronic Survey Methods.¹ Recruitment and marketing materials are reviewed at each survey cycle and new strategies are implemented, as appropriate and necessary

The Millennium Cohort team has undertaken countless efforts to ensure maximum response rates to surveys. The team consults with survey experts and new approaches are implemented and evaluated during each survey cycle. Each participant is sent paper questionnaires and/or directions to complete the questionnaire online in addition to email requests to participate depending on whether or not s/he has responded to the last questionnaire/email request sent. Each questionnaire mailing/email-request-to-participate is followed at approximately two weeks later by a reminder postcard/reminder email. Our approach of contacting the participants over a longer period of time (approximately 18 months) is conducted in case the participant is deployed or has recently moved to allow time for the current address to be updated. If a participant has not responded after 3 such invitations and we have not received notice that the address at which we attempted to reach them was not valid we infer that the questionnaire/email-request was received by the participant and that they have chosen not to participate. Such individuals are classified at that time as nonresponders. The protocol states that 3 attempts be made to rectify “bad” addresses. New or “good” addresses are sought from the Internal Revenue Service through contract with the National Institute for Occupational Safety and Health (NIOSH), Defense Manpower Data Center (DMDC), as well as self-reported respondent updates.

Response and retention rates are of utmost importance to Millennium Cohort and Family Study investigators. Much effort has been focused on investigation of the type and the use of incentives, wording of invitations, email contact, and twice yearly postcard and email contact on Veteran’s Day and Memorial Day (Millennium Cohort Study), or Month of the Military Child and National Military Month (Family Study). Past incentives included specially designed t-shirts with study logos, and phone cards. For the 2011-13 survey cycle effort, incentives included a $5 gift card (Starbucks, Subway, Amazon.com, WalMart), or a Millennium Cohort hat or coin. The Family Study offered $10 incentives for survey completion from Starbucks, Subway, or Shutterfly.

Invitations and email contacts are designed specifically for service branch, separation status, and other demographics and vetted through study team members. Twice yearly contact using Veteran’s Day and Memorial Day (Millennium Cohort Study), or Month of the Military Child and National Military Month (Family Study) serve to keep the service members and spouses engaged as well as solicit change of contact information. Retention centers around a website with sections dedicated to keeping those who have left military service in the cohort. Additionally, the 2007-08 enrollment cycle deployed a “welcome to the cohort” campaign that sent welcome cards to enrolling cohort members describing the scope and length of the study once again.

The first telephone survey of non-responders was performed after the Millennium Cohort Pilot Study in 2000 and focused on survey content and reasons for non-response. More recently, in 2005, a telephone survey of non-responders was conducted during the 2004-2006 data collection effort. The submitted report, “MilCohort Non-response Study Final Report”, describes the Millennium Cohort Telephone Study of 3,000 non-responders conducted by the Research Triangle Institute (RTI). This sub-study consisted of telephone calls to Panel 1 participants who had not completed a 2004-2006 questionnaire and resulted in a 31% response rate. In addition to asking questions regarding reasons for nonresponse, the phone survey asked about incentives, participant contact, and collected information on health status. For those with bad phone numbers (e.g., disconnected, wrong number), RTI completed a thorough investigation to obtain up-to-date phone numbers. This additional contact information was given to the Millennium Cohort Study team after the completion of the RTI survey and report.

Chapter 3 of RTI's “MilCohort Non-response Study Final Report” discusses telephone questionnaire results and presents recommendations for improving response rates. Overall, recommendations covered six main areas: study materials, panel maintenance, tracking sample members, incentives, telephone prompting, and future non-response studies. Many of these suggestions were incorporated in the 2007-08 and 2011-12 survey cycle efforts. Study materials emphasized that participants had the option to complete surveys on the web or by paper, and were tailored to be service specific and based on current military status (i.e., separated/retired or still serving). The continued use of bi-annual postcards and the utilization of the National Change of Address Service through the US Postal Service enhanced panel maintenance and the tracking of sample members. Incentives were offered this past survey cycle, to participants that completed their survey on the web. Participants were able to choose either a $5 gift card to one of four popular vendors, a commemorative challenge coin, or a ball cap with the study logo. In addition, automated telephone messaging was used among consented participants to encourage them to complete the survey. As a result of these additional marketing strategies during the 2007-08 cycle, 8,259 Panel 1 participants completed their second follow-up survey that did not submit their first follow-up survey.

Nearing the end of the survey cycle, expected to be June 2015, we will conduct a non-response survey among Panels 1-4 non-responders to ask about their impressions of the study, reasons for not participating, and about their general health. These data will be utilized in the design of the next survey cycle to maximize retention. The survey will be finalized based on survey response data from the upcoming survey cycle. An early draft of the proposed survey has been included as a reference as part of this submission. A separate Information Collection Request will be submitted to OMB for this non-response survey.

Longitudinal studies, such as the Millennium Cohort Study, provide the capability to prospectively analyze relationships between exposures and outcomes. A cohort study of military Service members and Veterans over an extended period of time provides the unique opportunity to examine the temporal relationships between service-related experiences, including deployment, with subsequent health and behavioral outcomes, which is not possible using time-series or cross-sectional samples. However, in longitudinal studies, non-response and non-random attrition is a potential source of bias. Analysis based on multi-wave panel studies can be heavily compromised by non-random sample attrition. While Millennium Cohort participants are given the opportunity to complete follow-up questionnaires regardless of whether they completed the previous follow-up questionnaire, the number of responders who do not participate in each subsequent wave of data collection (wave non-response) will most likely accumulate over time, which may undermine the precision of any research undertaken using such samples. Unless non-response is random, attrition may lead to bias, as there are important factors that influence response propensity. Attrition is often correlated with observable characteristics such as age, education, health, and economic well-being, as well as other unknown or unobserved factors. This non-random attrition can result in samples that include only a selected group of individuals over time, which can bias estimates since the non-response can often be associated with the variables of interest. However, non-random attrition does not necessarily lead to attrition bias. Attrition bias is model-specific and, as previous studies have shown, biases might be absent even if attrition rates are high.

The Millennium Cohort team has previously examined potential bias from initial enrollment and attrition. Demographic data on all invited personnel have been examined to determine differences in distributions among responders and non-responders. While overall the investigations have demonstrated the responders to be demographically representative of the invited sample², some factors have been found to be associated with greater likelihood of enrollment, including some demographic and military characteristics. While some of these differences are significant, most of them are quite small and are similar to patterns found in other surveys of military and non-military populations such as MIDUS and NHANES. These investigations have informed our statistical weighting techniques. Similar to other national studies using probability sampling and weighting to the Current Population Survey, the Millennium Cohort Study weights the study to the entire US military population using DMDC records. In addition to creating sampling weights to create robust standard errors to account for the complex survey design, we will also create non-response and attrition weights using multivariable multinomial logistic regression to calculate the propensity for response, non-response, or death. Details on these weighting procedures are described below in sections A and B. Further examination found few health differences between Millennium Cohort responders and non-responders when comparing healthcare utilization preceding study invitation.^3,4 In addition, non-response to the follow-up questionnaires has not resulted in any appreciable biases as reflected by comparing measures of association for selected outcomes, including PTSD, depression, and eating disorders, using complete case and inverse probability weighted methods.⁵

However, although previous work has indicated minimal bias has been introduced into the Millennium Cohort Study due to initial and follow-up non-response, continued investigation of factors leading to non-random attrition is key for adjusting for non-response and producing unbiased estimates. Most researchers agree weighting is necessary in descriptive analyses and creating population estimates, however, weighting in multivariable analyses is debatable.

The Millennium Cohort Study research team has previously calculated response rates, created sample and design weights, and has conducted initial response bias investigations for each panel as well as a few select longitudinal response bias investigations. Moving forward, the study team will systematically perform the following procedures to investigate and adjust for non-response bias for each panel and each completed wave (see attached table for related projects).

A. Procedures to Investigate Non-response for Panels at Baseline:

1. Response rates will be calculated using standard formulas using OMB Standards and Guidelines for Statistical Surveys (2006). For example:

• Unweighted unit response rates (RRU) as the proportion of those that were eligible for the survey at baseline that responded.

2. Non-response bias will be estimated and described by comparing responders to non-responders based on variables available from electronic personnel files available on all Service members via Defense Manpower Data Center. These variables will include factors such as demographic characteristics, deployment histories, and medical/healthcare data. For example, methods to do this will include, but will not be limited to, multivariable logistic regression to describe the propensity to respond or not respond.

3. Weights will be calculated using the propensity score for response, in order to minimize response bias. These non-response weights will be used in future studies, where appropriate.

B. Procedures to Investigate Non-response and Attrition for Panels at Follow-Up:

1. Response rates will be calculated using standard formulas using OMB Standards and Guidelines for Statistical Surveys (2006). For example:

• Longitudinal response rates as the proportion of responders at wave 1 (baseline), who responded at a specific subsequent wave.

2. Non-response and attrition bias will be examined and described by comparing responders to non-responders based on previous survey data. These variables will include factors obtained at the baseline survey and beyond, such as behavioral, mental, and physical health factors, as well as demographic characteristics, deployment histories, and medical/healthcare data, and in certain cases response to previous waves. For example, methods to do this will include, but will not be limited to, multivariable multinomial logistic regression to calculate the propensity for response, non-response, or death. We will separately model attrition due to death (or attrition by other causes, when appropriate) versus non-response since determinants will likely differ between these groups and may be differentially associated with outcomes of interest.

3. Attrition and non-response weights will be calculated based on the inverse of the wave specific probability of responding at that wave. Depending on specific longitudinal study design, the study team may utilize the baseline weight, the terminal wave weight, or the product of the wave-specific weights, noting that choosing the baseline weight or terminal wave weight may not properly adjust for attrition or differential left censoring.⁶

Given the evolving complexity of the Millennium Cohort Study methodology, we will consult with an experienced survey methodologist with expertise in creating and applying survey weights for longitudinal studies with complex sampling methods. Weights will be available as variables in all datasets. Documentation of the methodology used to create these weights will be publicly available as a downloadable document on the Millennium Cohort Study website (http://www.millenniumcohort.org).

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Following preliminary focus group evaluations of the draft questionnaire conducted in late 1999 with military enlisted and officer groups of less than 10 people, a pilot study was conducted on a 1% sample of military personnel in the spring of 2000 as a means of testing the utility of the instrument. Following this pilot, corrections were made to produce the final survey instrument.

Along with non-response testing as described earlier in the report, we will conduct focus group testing. The purpose of the focus group testing is to determine effective strategies to maximize participation rates in populations with similar demographics to Millennium Cohort participants, ensure the clarity of our contact materials, including the text and overall visual format of emails, postcards, and survey packets, and lastly, to obtain feedback on the cost-saving initiatives currently offered.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

**************************

Martin White, MPH, interim Principal Investigator and Department Head, Department of Deployment Health Research, Naval Health Research Center.

[email protected]

(619) 553-9292

For the Family Study

William E. Schlenger, Ph.D. Principal Scientist,

Representative for the Research Team at Abt Associates

5001 South Miami Blvd, Suite 210, Durham, NC 27703

[email protected] (919) 294-7712

www.abtassociates.com

**************************

The Research Team at the Department of Defense Center for Deployment Health Research, Naval Health Research Center, San Diego CA 92106-3521

[email protected]

(619) 553-7465

References

1. Dillman DA. Mail and Telephone Surveys: The Total Design Method. New York: Wiley; 1978. xvi, 325.

2. Ryan MA, Smith TC, Smith B, Amoroso P, Boyko EJ, Gray GC, Gackstetter GD, Riddle JR, Wells TS, Gumbs G, Corbeil TE, Hooper TI, for the Millennium Cohort Study Team. Millennium Cohort: enrollment begins a 21-year contribution to understanding the impact of military service. Journal of Clinical Epidemiology. 2007 Feb;60(2):181-91.

3. Wells TS, Jacobson IG, Smith TC, Spooner CN, Smith B, Reed RJ, Amoroso PJ, Ryan MAK, for the Millennium Cohort Study Team. Prior health care utilization as a determinant to enrollment in a 22-year prospective study, the Millennium Cohort Study. European Journal Of Epidemiology. 2008 Feb;23(2):79-87.

4. Horton JL, Jacobson IJ, Littman AJ, Alcaraz JE, Smith B, and Crum-Cianflone NF. The impact of deployment experience and prior healthcare utilization on enrollment in a large military cohort study. BMC Medical Research Methodology. 2013 Jul 11;13:90.

5. Littman AJ, Boyko EJ, Jacobson IG, Horton JL, Gackstetter GD, Smith B, Hooper TI, Amoroso PJ, Smith TC, for the Millennium Cohort Study Team. Assessing non-response bias at follow-up in a large prospective cohort of relatively young and mobile military service members. BMC Medical Research Methodology. 2010 Oct;10(1):99.

6. Hill DH. Adjusting for attrition in event-history analysis. Sociological Methodology 1997; 27:393-416.

10