Section B. Collection of Information Employing Statistical Methods
B.1.a. Respondent Universe and Sampling Method
The target population for this cross-sectional survey consists of active duty service members who are aged 20-to-24 years three months prior to the expected start date of the survey data collection. This target population contains two primary groups of interest: (1) separated regular service members age 20-to-24 years, and (2) separated (i.e., demobilized) National Guard and Reserve members aged 20-to-24 years. Only veterans living in households at the time of the survey will be considered eligible for the survey. The sample will be drawn by the Defense Manpower Data Center (DMDC) from their database and provided to the Veterans Administration. DMDC maintains up-to-date information on all active duty, Guard, and Reserve service members. The survey will be conducted primarily by telephone.
The study is being design and conducted with the assistance of a contractor:
Contractor: Abt Associates Inc.
Suite 800 North
4550 Montgomery Ave
Bethesda, MD 20814
Contractor Roles:
Questionnaire design and testing
Statistical sample design
Data collection
Data processing and weight calculations
Preparation of survey data file with documentation and a survey methodology report.
B.1.b. Feasibility
The feasibility of the design rests primarily on the ability to obtain a national probability sample of separated regular service, National Guard, and Reserve members aged 20-to-24 years three months prior to the expected start of the survey. Fortunately, a list of this population is maintained by the DMDC. Their database is updated on a monthly basis and contains the name, address, separation date and other relevant information for each individual separated from active duty. A second aspect of the feasibility of the design relates to the ability of Abt Associates to locate and contact the members of the sample. The DMDC database has information on the last known home address of the individual, which will serve as a starting point for locating a current telephone number. An address match with the TARGUSinfo, Inc. data base of residential addresses and telephone numbers may yield, based on our experience using this procedure, telephone numbers for as high as 65-70 percent of the sample. For younger veterans, the last known home address and telephone number is more likely to be that of the parents. It is likely that, on many occasions, we will try to obtain a new address and/or telephone number from the parents, because the sampled veteran has moved. Also, if the address match rate produces telephone numbers for well below 65-70 percent of the sample, it may be necessary to modify the data collection approach to include multiple modes of data collection to avoid the potential for noncoverage bias. The easiest mode to use for an individual without a telephone number is a mail questionnaire. In these instances, a shortened version of the questionnaire may be more appropriate for a mail survey, so copies of a shortened questionnaire will be mailed to these potential respondents.
The recruiting phase is particularly important because most non-response occurs at this point with the potential for introducing non-response bias.
To introduce the study to sampled veterans, two advance letters will be sent in one packet: one from the VA introducing the study and another letter from the Abt Associates Project Director providing details of the telephone survey as well as the 1-800-number so that potential respondents can contact us.
B.2.a. Target Population and Coverage
The target population for this cross-sectional survey consists of active duty service members who are aged 20-to-24 years three months prior to the expected start date of the survey data collection. This target population contains two primary groups of interest: (1) separated regular service members age 20-to-24 years, and (2) separated National Guard and Reserve members aged 20-to-24 years. Only veterans living in households will be considered eligible for the survey. The sample will be drawn by the Defense Manpower Data Center (DMDC) from their database, which offers 100 percent coverage of the target population. As of August 2006 the population consists of 156,228 separated regular service members and 64,389 separated National Guard and Reserve members.
B.2.b. Sample Design
The desired number of completed interviews is approximately 2,000. To obtain separate estimates for regular service and National Guard and Reserve veterans aged 20-to-24 years who were separated, the sample will be stratified on this variable, and approximately 1,000 interviews will be allocated to each of these two primary strata.
B.2.b.1. Sampling Frame
As noted above, the sample will be drawn by the Defense Manpower Data Center (DMDC) from their database, which offers 100 percent coverage of the target population. This database contains the following key variables:
Street Address, City, State and ZIP Code
Race
DOB/Age
Gender
Military Occupation Specialty (MOS) (most recent)
Served on Active Duty
Branch of Service
Armed Forces Qualification Test scores (AFQT)
Date of Separation
Officer/Enlisted
Period of Service
Rank
Disability rating and year received
Combat (wars and theaters).
B.2.b.2. Sample Size
As noted earlier in this section, the desired number of completed interviews is approximately 2,000. To obtain separate estimates for active duty regular service and National Guard and Reserve veterans aged 20-to-24 years who were separated, the sample will be stratified on this variable and approximately 1,000 interviews will be allocated to each of these two primary strata. The initial sample size will be around 5,000 members from each primary stratum. A total sample size of 10,000 members is considerably larger than we expect to need to reach the target number of interviews, but we prefer to have a larger than needed sample as a way to guard against the possibility that one of the sample size assumptions will not hold and that additional sample will be needed quickly. This sample will be divided into sample replicates, and, for survey implementation, only the replicates required to attain around 1,000 completed interviews from a primary stratum will be released.
B.2.b.3. Stratification Variables
Secondary stratification within each primary stratum will be used to ensure a well-distributed national probability sample. Secondary stratification variables include:
Race/Ethnicity
Gender
Officer versus Enlisted
Branch of Service.
B.2.b.4. Sampling Method
For each primary stratum, a proportionate stratified sample design will be used. The Veterans Administration and Abt Associates will provide sampling specifications to the DMDC. We will work with them to either draw a proportionate stratified sample using simple random sampling within each secondary stratum. As discussed below a proportionate stratified sample allocation is expected to yield a sufficient number of completes questionnaires for most key subgroups. The only departure that we expect from an exact proportionate stratified sample is to oversample two very small subgroups. First, we will include all 39 members of the Naval Reserve in the sample. Second, we will include 100 of the 389 officers in the sample. This slight departure will have very little impact on the precision of the overall estimates and will allow us to have a sufficient number of completes questionnaires to examine these two very small subgroups more from a qualitative point of view.
Most of the analysis will examine regular service veterans separately from National Guard/Reserves veterans. For an estimate of a population proportion around 0.50, the expected 95 percent confidence interval half-width is 0.033. Much of the analysis will focus on subgroups (for example, estimates for race groups, by gender, and for military occupation specialty categories). Using a 95 percent confidence interval half-width of 0.10 as reasonable level of precision for subgroups, a subgroup sample of around 150 completed questionnaires would be required. It is likely that estimates can be produced for the following key subgroups (expected number of completed questionnaires based on the August 2006 data base is shown in parentheses):
Enlisted (1,997)
Male (1,646)
Female (354)
Less than one year of active service (887)
One year of more but less than two years of active service (484)
Two years of more but less than four years of active service (288)
For or more years of active service (340)
Separated for less than 12 months (641)
Separated for 12 months to less than 24 months (431)
Separated for 24 months or more (928)
Non-Hispanic white (1,310)
Non-Hispanic black (271)
Hispanic (188)
Army (422)
Navy (254)
Marine Corps (230)
Army National Guard (516)
Army Reserve (254)
Marine Corps Reserve (142)
Air Force/Air Force Reserve/Air National Guard (182)
B.2.c.1. Nonsampling Error
After the cross-sectional survey has been completed, we will develop sampling weights that include an internal unit non-response adjustment. While a variety of methods will be used to maximize response, some degree of non-response is inevitable, and the final AAPOR #3 response rate1 is expected to be approximately 80 percent. We plan to include a non-response bias study. It is fortunate that the sampling frame contains a wide variety of variables. We expect that the database from the DMDC will include:
Race
DOB/Age
Gender
Military Occupation Specialty (MOS) (most recent)
Served on Active Duty
Branch of Service
Armed Forces Qualification Test scores (AFQT)
Date of Separation
Officer/Enlisted
Period of Service
Rank
Disability rating and year received
Combat (wars and theaters).
The sampling frame variables will be used in a two-part non-response bias analysis. For the cross-sectional survey—for each of the two primary strata (regular service and National Guard/Reserve) discussed above, we will first compare characteristics of individuals for which accurate locating/contact information was obtained with individuals for whom information could not be obtained. The second comparison will be limited to persons in the sample for which accurate locating/contact information was obtained. We will compare information on individuals who complete interviews with information on persons who do not complete interviews. The results of the non-response bias analysis will also be used to identify sampling frame variables that will be used to construct marginal population control totals for raking. The use of raking to population control total margins will yield the final weight for use in estimation. Standard errors will be estimated using SUDAAN to reflect the use of weights and the sample design in the standard error calculations.
B.3. Methods to Maximize Response Rates
Response rates are a crucial issue in surveys. While a variety of methods will be used to maximize responses, some degree of non-response is inevitable, especially in a quick response survey. Based on our experience, Abt expects to obtain about an 80 percent response rate for the proposed study.
In an effort to achieve high response rates, Abt proposes the following:
Advance Letter and Introduction. Two versions of the initial recruiting letter and introduction used by the telephone interviewers will be tested to develop the most effective approach. Copies of these advance letters appear in Attachment C3.
Refusal Aversion/Refusal Conversion. Telephone interviewers are trained to understand respondent concerns about confidentiality or other reasons for reluctance to participate. Interviewer training includes materials on gaining cooperation; interviewers are also provided with a set of procedures and systems supporting non-response followup and refusal conversion. These systems and procedures include:
An ongoing interviewer training program in refusal aversion to reduce the number of unresolved cases and refusals to participate.
Use of advance mailings described above.
For refusal conversion specialists, a training module that stresses the importance of professional courtesy, familiarity with the project’s goals and the steps in the recruiting process, and effective techniques for allaying respondent concerns. These specialists work with telephone center supervisors to create case-specific strategies to encourage participation.
Abt Associates will attempt to minimize non-response at each stage of this survey. Procedures are described below.
Apply a call management system that tracks and manages the sample of telephone numbers so that telephone numbers are called at different times on different days; that reminds interviewers so appointments with respondents are kept; and that insures callbacks are made at the appropriate times.
Maintain a sufficient staff of interviewers so that respondents are called in an efficient and timely manner—appointments and callbacks must occur at the correct times even during peak calling hours.
Train interviewers in refusal aversion techniques to prevent initial refusals.
Perform on-line monitoring of 5 percent of all calls placed so that action can be taken to correct poor interviewing practices.
Identify best interviewing practices and share them with the entire interviewing staff through regular project meetings and refresher training.
Arrange ongoing training for interviewers and supervisors to improve their skills and alert them to protocol changes and revisions.
While it is well-known that response rates, in general, are declining, especially in telephone surveys, Abt Associates (VA’s contractor for this study) has a proven record of maintaining high response rates over time. Abt Associates expects to obtain accurate locating/contact information for about 80 percent of the sample. For individuals with accurate locating/contact information, it is expected that the survey procedures described above will yield an interview completion rate around 90 percent. The survey’s goal is an overall response rate around 80 percent.
B.3.c. Proposed Non-response Analysis
As described above, a variety of methods will be used to maximize response. Nonetheless, some degree of non-response is inevitable. This data collection includes a non-response bias study. It is fortunate that the sampling frame contains a wide variety of variables. The database from the Defense Manpower Data Center (DMDC) will include:
Address
Race
DOB/Age
Gender
Military Occupation Specialty (MOS) (most recent)
Served on Active Duty
Branch of Service
Armed Forces Qualification Test scores (AFQT)
Date of Separation
Officer/Enlisted
Period of Service
Rank
Disability rating and year received
Combat (wars and theaters).
These sampling frame variables will be used in the following non-response bias analysis. First, for each of the primary and secondary strata, we will compare the characteristics of sample individuals for which accurate locating/contact information was obtained with individuals for whom the information could not be obtained. Second, for sample individuals with accurate locating/contact information, we will compare the characteristics of those individuals who complete the survey versus those individuals who fail to complete the survey. Based on the findings of these analyses, specific sampling frame variables that exhibit differences will be used in the poststratification of the sample of respondents to population control totals.
Pre-testing was discussed in Section A. A pre-test will ensure that the draft instrument performs properly under actual telephone interview and web conditions. A replicate (or subsample) of 50 cases will be fielded. Two interviewer debriefing sessions—one after the pretest and one shortly into main survey data collection—will be conducted to help identify issues to be addressed prior to conducting the pilot study.
B.5. Statistical Consultants
The Abt Associates statistical consultants listed below will be available to the VA.
Johnny Blair
Principal Scientist and Senior Survey Methodologist
(301) 634-1825
Michael Battaglia
Vice President, Principal Scientist, Senior Sampling Statistician
(617) 349-2425
1 Response Rate 3 (RR3) estimates what proportion of cases of unknown eligibility is actually eligible. For a discussion of calculating outcome rates see Standard Definitions Final Dispositions of Case Codes and Outcome Rates for Surveys, The American Association for Public Opinion Research, 2006.
| File Type | application/msword |
| File Title | JUSTIFICATION OF EVIDENCED-BASED COMMUNICATIONS STRATEGIES |
| Author | Carolyn Holmes |
| Last Modified By | Denise McLamb |
| File Modified | 2006-12-21 |
| File Created | 2006-12-21 |