0704-0615_wgr_ssb_20240408

Supporting Statement Outline – Sample

NOTE: Complete Part B for Survey ICR Requests

SUPPORTING STATEMENT – PART B

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

If the collection of information employs statistical methods, the following information should be provided in this Supporting Statement:

1. Description of the Activity

Describe the potential respondent universe and any sampling or other method used to select respondents. Data on the number of entities covered in the collection should be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate the expected response rates for the collection as a whole, as well as the actual response rates achieved during the last collection, if previously conducted.

The WGR surveys will use a single-stage non-proportional stratified random sampling to identify eligible DoD participants in order to achieve precise estimates for important reporting categories (e.g., gender, Service, pay grade). The total sample size for the Active component will be approximately 735,000 (including a census of the Coast Guard) and is based on precision requirements for key reporting domains (prevalence rates by Service, gender, & pay grade). Given anticipated eligibility and response rates, an optimization algorithm will determine the minimum-cost allocation that simultaneously satisfies the domain precision requirements. Anticipated eligibility and response rates for the Active component will be based on the 2023 WGR of Active Duty Members. However, the 2023 WGR is still under final review and analysis, so 2021 WGR response rates are shown in Table 1 for illustration purposes.

The total sample size for the Reserve component will be approximately 250,000 (including a census of the Coast Guard Reserve) and is based on precision requirements for key reporting domains. Given anticipated eligibility and response rates, an optimization algorithm will determine the minimum-cost allocation that simultaneously satisfies the domain precision requirements. Anticipated eligibility and response rates for the Reserve component will be largely based on the 2023 WGR of Reserve Component Members. However, the 2023 WGR is still under final review and analysis, so 2021 WGR response rates are shown in Table 2 for illustration purposes.

Because precise estimates on the percentage of reported sexual assault rates among even the smallest domains (e.g., Marine Corps women) are required, a sizable sample is necessary.

Table 1.

2021 Active Component Sample Size by Stratification Variables

Variable	Total	Army	Navy	Marine Corps	Air Force	Coast Guard
Sample	746,987	265,387	191,147	115,081	135,090	40,282
Gender
Male, Unknown	558,803	207,330	136,216	98,776	82,423	34,058
Female	188,184	58,057	54,931	16,305	52,667	6,224
Paygrade Group
E1–E4	419,665	157,624	103,466	83,155	62,905	12,515
E5–E9	237,477	75,753	68,915	22,941	50,713	19,155
W1–W5	7,718	4,694	572	779	0	1,673
O1–O3	54,036	18,925	12,589	5,383	13,060	4,079
O4–O6	28,091	8,391	5,605	2,823	8,412	2,860
Race
Non-Minority	413,166	140,744	98,714	68,454	75,942	29,312
Minority	333,821	124,643	92,433	46,627	59,148	10,970

Table 2.

2021 Reserve Component Sample Size by Stratification Variables

Variable	Total	Army National Guard	Army Reserve	Navy Reserve	Marine Corps Reserve	Air National Guard	Air Force Reserve	Coast Guard Reserve
Sample	247,839	76,269	55,725	26,182	25,162	26,445	31,848	6,208
Gender
Male/Unknown	146,192	43,483	29,878	12,034	23,601	14,104	17,946	5,146
Female	101,647	32,786	25,847	14,148	1561	12,341	13,902	1,062
Paygrade Grouping
E1-E4	117,919	45,788	28,563	6,362	16,694	8,777	10,620	1,115
E5-E9	80,703	16,365	15,046	14,177	4,220	12,231	14,583	4,081
W1-W5/O1-O3	28,770	11,753	7,748	2,107	1676	2,674	2,279	533
O4-O6	20,447	2,363	4,368	3,536	2572	2,763	4,366	479
Reserve Program
TPU	201,052	65,485	47,508	21,323	21,314	17,903	21,311	6,208
AGR	18,238	4,557	3,062	4,788	1288	3,363	1180	0
MilTech	17,287	6,227	2,839	0	0	5,179	3,042	0
IMA	11,262	0	2316	71	2560	0	6,315	0

The potential respondent universe is the entire Active and Reserve component. In these cases, a no more than ten question abbreviated version of the WGR survey will be open to all military members (Active and Reserve components) who are not in the study sample. In other words, OPA will allow military members who learn about the survey and wish to participate, but were not randomly selected into the study sample, to have the option to respond to a limited set of questions regarding their workplace. In fact, providing this option has the broader goal of ensuring that Service members who wish to share their experiences vis-à-vis their workplace have a means to do so. Data from “out of sample” respondents will not be combined with the sampled member’s responses for weighting and analysis purposes. Instead, responses from “out of sample” respondents may be used in separate analyses of Service member perceptions of organizational climate.

2. Procedures for the Collection of Information

Describe any of the following if they are used in the collection of information:

1. Statistical methodologies for stratification and sample selection;

Within each stratum, individuals will be selected with equal probability and without replacement. However, because allocation of the sample will not be proportional to the size of the strata, selection probabilities will vary among strata, and individuals will not be selected with equal probability overall. Non-proportional allocation will be used to achieve adequate sample sizes for small subpopulations of analytic interest for the survey reporting domains. These domains will include subpopulations defined by the stratification characteristics, as well as others.

The OPA Sample Planning Tool, Version 2.1 (Dever and Mason, 2003) will be used to accomplish the allocation. This application will be based on the method originally developed by J. R. Chromy (1987), and is described in Mason, Wheeless, George, Dever, Riemer, and Elig (1995). The Sampling Tool defines domain variance equations in terms of unknown stratum sample sizes and user-specified precision constraints. A cost function is defined in terms of the unknown stratum sample sizes and per-unit costs of data collection, editing, and processing. The variance equations are solved simultaneously, subject to the constraints imposed, for the sample size that minimizes the cost function. Eligibility rates modify estimated prevalence rates that are components of the variance equations, thus affecting the allocation; response rates inflate the allocation, thus affecting the final sample size.

2. Estimation procedures;

The eligible respondents will be weighted in order to make inferences about the entire active duty or Reserve component populations. The weighting will utilize standard processes. The first step will be to assign a base weight to the sampled member based on the reciprocal of the selection probability. For the second step, non-response, the base weights will be adjusted (eligibility and completion of the survey). Finally, the current weights will be post-stratified to known population totals to reduce bias associated with the estimates. Variance strata will then be created so precision measures can be associated with each estimate. Estimates will be produced for reporting categories using 95% confidence intervals with the goal of achieving a precision of 5% or less.

3. Degree of accuracy needed for the Purpose discussed in the justification;

OPA allocated the sample to achieve the goal of reliable precision on estimates for outcomes associated with reporting a sexual assault (e.g., retaliation) and other measures that were only asked of a very small subset of members, especially for males. Given estimated variable survey costs and anticipated eligibility and response rates, OPA used an optimization algorithm to determine the minimum-cost allocation that simultaneously satisfied the domain precision requirements. Response rates from previous surveys were used to estimate eligibility and response rates for all strata.

The allocation precision constraints will be imposed only on those domains of primary interest. Generally, the precision requirement will be based on a 95% confidence interval and an associated precision of 5% or less on each reporting category. Constraints will be manipulated to produce an allocation that will achieve satisfactory precision for the domains of interest at the target sample size of approximately 735,000 members for the Active component and approximately 250,000 for the Reserve component.

4. Unusual problems requiring specialized sampling procedures; and

None.

5. Use of periodic or cyclical data collections to reduce respondent burden.

The WGR surveys occur on a biennial basis (i.e. every other year) in accordance with the congressional mandate.

3. Maximization of Response Rates, Non-response, and Reliability

Discuss methods used to maximize response rates and to deal with instances of non-response. Describe any techniques used to ensure the accuracy and reliability of responses is adequate for intended purposes. Additionally, if the collection is based on sampling, ensure that the data can be generalized to the universe under study. If not, provide special justification.

Multiple reminders will be sent to sample members during the survey field period to help increase response rates. Both postal letters and e-mails will be sent to sample members until they respond or indicate that they no longer wish to be contacted. The outreach communications will include text highlighting the importance of the surveys. Additional outreach informing all Service members of the survey fielding is planned via military websites (e.g., Military OneSource), platforms (e.g., SAPR Connect), and social media (e.g., Instagram, Twitter, & Facebook).

The sample sizes will be determined based on prior response rates from similar active duty or Reserve component surveys. Given the anticipated response rates, OPA predicts enough responses will be received within all important reporting categories to make estimates that meet confidence and precision goals. A non-response bias study will be conducted to compare known values to weighted estimates from the survey and analyze missing data/drop-off.

4. Tests of Procedures

Describe any tests of procedures or methods to be undertaken. Testing of potential respondents (9 or fewer) is encouraged as a means of refining proposed collections to reduce respondent burden, as well as to improve the collection instrument utility. These tests check for internal consistency and the effectiveness of previous similar collection activities.

In accordance with the DoD Survey Burden Action Plan, we went to great lengths to develop a survey instrument that collected the information required to meet the Congressional-mandate and to support policy and program development and/or assessments. To accomplish this, we reviewed data from the 2018 WGR survey of the Active component and 2019 WGR survey of the Reserve component to identify items that did not perform well, for which very little usable data were collected, or that did not support information requirements. This process allowed us to delete 42 items from the survey in 2021, and was used again in 2023. We also collaborated with leaders from each of the relevant policy offices (i.e. DoD Sexual Assault Prevention and Response Office [SAPRO], the Office for Diversity, Equity, and Inclusion [ODEI], & the Office of Force Resiliency [OFR]) to identify their critical information needs and research questions. This led us to identify additional items to remove from the survey as well as important gaps in knowledge regarding response to, and especially prevention of, unwanted gender related behaviors that additional survey items could address—namely, additional information regarding bystander intervention, leader behaviors, and the prevalence of sexist beliefs. We were ultimately able to remove an additional 67 items from the survey, even after adding survey items to address information gaps.

New or revised survey items reflect the DoD’s renewed focus on the prevention of unwanted gender-related behaviors. More specifically, the survey instrument replaces some items included in prior years (e.g., questions about training) with items specifically designed to support prevention (e.g., sexual harassment and stalking items, additional bystander intervention items, and items related to sexist beliefs). In all cases, a review of the literature was used to identify peer-reviewed and validated scales for use in the survey instrument (where available).

The WGR of Military Members involves complex skip patterns that substantially reduce the number of questions respondents will be asked based on their answers to previous questions. Because of these skips, we anticipate that the vast majority of respondents will NOT see all of the follow-up questions that are included in several sections of the survey (e.g., questions asking respondents to describe the characteristics of a sexual harassment or unwanted sexual contact experience). The survey includes a total of 120 questions. However, based on skip logic, the vast majority of respondents (~75%) will only see 50 questions.

5. Statistical Consultation and Information Analysis

a. Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

Mr. Matthew Scheidt, Statistics Team Lead, OPA, [email protected]

b. Provide name and organization of person(s) who will actually collect and analyze the collected information.

Data will be collected by Data Recognition Corporation, OPA’s operations contractor. Contact information is listed below.

• Ms. Valerie Waller, Senior Managing Director, Data Recognition Corporation, [email protected]

Data will be analyzed by OPA social scientists & analysts. Contact information is listed below.

• Dr. Jessica Marcon Zabecki, Chief Military Research Psychologist, Health & Resilience (H&R) Research Division – OPA, [email protected]

• Dr. Ashlea Klahr, Acting Director of OPA, [email protected]

• Dr. Rachel Lipari, Chief Military Sociologist, H&R Research Division – OPA, [email protected]

• Ms. Lisa Davis, Deputy Director, H&R Research Division – OPA, [email protected]

• Dr. Hannah Samuelson, Senior Research Psychologist, H&R Research Division– OPA, [email protected]

• Ms. Kimberly Hylton, Chief of Survey Operations and Methodology, H&R Research Division – OPA, [email protected]

• Dr. Christine McCall, Operations Analyst – Fors Marsh, [email protected]

• Ms. Alycia Polant, Operations Analyst – Fors Marsh, [email protected]

• Ms. Jessica Tercha, Associate Director -- Fors Marsh, [email protected]

• Ms. Amanda Barry, Director, Military Health & Wellbeing Research – Fors Marsh, [email protected]

• Ms. Margaret H. Coffey – Fors Marsh, [email protected]