Installation Evaluation Methodology

Installation Evaluation Methodology
Points of Contact
Office of Force Resiliency (OFR) Project Team:
Lead: Dr. Andra Tharp at [email protected]
Travis Bartholomew
Advana Project Team:
Lead: James Yamanaka at [email protected]
Melissa Macasieb
Daniel Dockterman
Jason Doering

Background
Secretary of Defense issued Memorandum, “Immediate Actions to Counter Sexual Assault and
Harassment and the Establishment of a 90-Day Independent Review Commission on Sexual Assault in
the Military,” February 26, 2021, which directed immediate actions to address sexual assault and
harassment. Immediate Action 2 directed OSD to conduct Installation Evaluations and to provide
quarterly command climate updates.
To support identification of installations for the FY21 evaluations, USD(P&R) directed a force-wide
Defense Organizational Climate Survey (DEOCS) to be completed. Command climate updates in FY22
and beyond, as well as subsequent installation evaluations will leverage additional data sources to
identify locations. The DEOCS was selected as the primary data source for the FY21 installation
evaluations because it serves as the most timely and sensitive DoD-wide measure of command climate
and because other relevant data, such as the Workplace Gender Relations Surveys and Status of Forces
Surveys, were delayed due to COVID, which precluded timely data from those data sources being
included in the FY21 HRIE.
DEOCS 5.0 is comprised of 19 factors, 9 of which depict risk factors and 10 of which depict protective
factors for readiness detracting behaviors, such as sexual assault, harassment, and suicide. However, for
the purposes of this analysis, transformational leadership ratings, passive leadership ratings, and toxic
leadership ratings are treated as separate factors for the unit/organization leader, commander, and the
Senior NCO, if applicable. As a result, this analysis includes 22 total factors: 11 risk and 11 protective
(see Appendix).

Data Transfer
All DEOCS data files are produced through an automated process. Each time data files are transferred to
other systems, files are validated by confirming that record counts match; in addition, individual values
are compared to the original file for select number of registrations. All variables are verified to ensure they
are transferred properly and contain valid values.

Data Ingestion and Inclusion/Exclusion Criteria
We ingested DEOCS 5.0 data into Advana from the contractor across three data file transfers: JanuaryMarch data was comprised of 237,104 survey respondent, April data was comprised of 482,745
respondent, and May data was still pending ingestion as of the time of publishing. In total, we received
DEOCS 5.0 surveys from 719,849 respondents across 7,142 units (as of January-April 2021).
Table 1 shows the total survey counts by Component and Service branch.

Installation Evaluation Methodology

Table 1: DEOCS 5.0 Survey Respondents, by Component and Service Branch (as of
January-April 2021)
Active

Reserve

Total

Service Branch
Army

229,825

21,629

254,750

Navy

138,696

2,272

142,289

Air Force

125,916

3,138

130,946

85,034

3,204

88,238

--

34,183

34,183

Marine Corps
National Guard
Space Force

2,730

--

2,730

Coast Guard

8,428

164

8,592

Joint Service

--

--

9,814

DoD

--

--

48,307

590,629

64,590

719,849

Total

Matching Units with Installations. Using data collected by the Services, we were able to match 5,589
out of 7,142 units with their respective installations for a match rate of 78%.

Installations for On-Site Evaluation
We employ a multi-measure approach in identifying military installations that are outliers in terms of risk
and protective factors. The identified locations will undergo an on-site evaluation (methods for on-site not
included in this document).
Installation Protective and Risk Percentile Scores. Using DEOCS 5.0 data collected at the unit level,
we aggregate to the installation level using mappings provided by the Services. We then categorize
installations within Service branch according to their Protective Percentile Score and Risk Percentile
Score.1 This was useful for reducing the total number of installations in each Service into more
manageable groupings for closer inspection.
Computing Percentile Scores. We calculate Protective and Risk Percentile Scores in four steps. To
help illustrate this computational process, we use as an example, the survey results from the 399
respondents across five units (3 Army and 2 Air Force) at an anonymized installation, Installation X (see
Table 2).

Protective and Risk Percentile Scores were strongly negatively correlated across installations (r = -0.90). This result was
expected given that higher Protective Percentile Scores correspond to positive behaviors and lower Risk Percentile Scores
correspond to negative behaviors.
1

Installation Evaluation Methodology

Table 2: Survey Respondents Completing DEOCS 5.0 at Installation X (Example)
Unit Name

Number of
Respondents

Component

Service

Unit A

Active

Air Force

189

Unit B

Active

Air Force

105

Unit C

Reserve

Army

57

Unit D

Reserve

Army

27

Unit E

Reserve

Army

21

Total

399

Step 1: We compute an average unit score for each factor, ranging from -1 to 1, by weighting the
proportion of responses in each category. Specifically, each negative category for a protective factor is
assigned a value of -1 (e.g., non-cohesive organization, low connectedness, etc.), each neutral category
is assigned a value of 0 (e.g., neutral, moderate, etc.), and each positive category is assigned a value of
1 (e.g., cohesive organization, high connectedness, etc.). For risk factor scores, we use the opposite
coding structure: each negative category is assigned a value of 1 (e.g., frequent binge drinking, passive
NCO leadership etc.), each neutral category is assigned a value of 0 (e.g., some binge drinking, neutral,
etc.), and each positive category is assigned a value of -1 (e.g., no binge drinking, non-passive
leadership, etc.).2
Installation X (Example): One hundred eighty-nine respondents completed the survey in Unit A, the
most of any of the five units at Installation X. For the factor Cohesion, this unit had a non-cohesive score
of 12.2%, a neutrally cohesive score of 14.9%, and a cohesive score of 72.9%. As a result, the composite
Cohesion factor score for Unit A is 0.61 (-1* .122 + 0*.149 + 1*.729 = 0.61). We repeat this calculation for
all Protective and Risk factors for this unit as shown below in Table 3.

Table 3: Factor Score Calculation for Unit A at Installation X (Example)
Unit Name
Protective Factors
Cohesion
Connectedness
Engagement & Commitment
Fairness
Inclusion
Morale
Safe Storage for Lethal Means
Work-life Balance
Leadership Support (Immediate Supervisor)
Transformational Leadership (Commander)
Transformational Leadership (Senior NCO)
Risk Factors

2

Factor Response Category
A

B

C

72.9%

14.9%

12.2%

81.5%
78.8%
56.0%
69.8%
47.5%
65.7%
86.8%
77.7%
68.1%
66.0%

10.6%
15.0%
21.8%
14.1%
36.6%
3.4%
7.9%
11.6%
24.4%
31.0%

7.8%
6.2%
22.3%
16.1%
15.9%
30.9%
5.3%
10.7%
7.5%
3.0%

Factor Score

0.61
0.74
0.73
0.34
0.54
0.32
0.35
0.81
0.67
0.61
0.63

For factors with only two response categories, each positive category is assigned a value of 3 (e.g., no presence of racially
harassing behaviors, no presence of sexist behaviors) and each negative category is assigned a value of 1 (e.g., presence of
racially harassing behaviors, presence of sexist behaviors).

Installation Evaluation Methodology
Alcohol Impairing Memory
Binge Drinking
Stress
Passive Leadership (Commander)
Passive Leadership (Senior NCO)
Toxic Leadership (Immediate Supervisor)
Toxic Leadership (Senior NCO)
Racially Harassing Behaviors
Sexist Behaviors
Sexually Harassing Behaviors
Workplace Hostility

0.0%
6.7%
31.7%
8.5%
2.5%
8.5%
2.0%
19.0%
6.3%
24.9%
88.4%

2.8%
29.6%
-27.1%
33.0%
11.0%
30.7%
-----

97.2%
63.7%
68.3%
64.4%
64.5%
80.5%
67.3%
81.0%
93.7%
75.1%
11.6%

-0.97
-0.57
-0.37
-0.56
-0.62
-0.72
-0.65
-0.62
-0.87
-0.50
0.77

Note: Stress, Racially Harassing Behaviors, Sexist Behaviors, Sexually Harassing Behaviors, and Workplace Hostility do not have
neutral categories. Factor Scores range from -1 to 1. ‘A’ response is favorable for Protective factors and unfavorable for Risk
factors; ‘B’ response is neutral; ‘C’ response is unfavorable for Protective factors and favorable for Risk factors.

Step 2: Next, we weight and aggregate all unit-level factor scores to the installation-level according to the
number of DEOCS respondents in each unit.3 This process ensures that the responses of each survey
taker in an installation (regardless of unit) are allocated equal weight in the calculation of the overall factor
score of the installation.
Installation X (Example): As shown in Table 4, nine times as many Service members in Unit A
completed the DEOCS 5.0 as compared with Unit E (n=21). As a result, the factor score for Unit A was
weighted nine times as heavily as Unit E. Because of the way scores happen to be distributed across
units, the unweighted and weighted factor scores for Cohesion are equivalent (0.72). However, for
Alcohol Impairing Memory, the weighted factor score is considerably lower than the unweighted score (0.94 vs. -0.87).
Table 4: Unit Weights for Cohesion Factor at Installation X (Example)
Factor Score
(Unweighted)

Number of
Respondents

Unit
Weight

Factor Score
(Weighted)

Unit A

0.61

189

2.37

1.44

Unit B

0.89

105

1.32

1.16

Unit C

0.83

57

0.71

0.59

Unit D

0.87

27

0.34

0.29

Unit E

0.38

21

0.26

0.10

Installation X Cohesion Factor Score

0.72

1.00

0.72

Unit Title
Cohesion

Alcohol Impairing Memory
Unit A

-0.97

189

2.37

-2.30

Unit B

-0.99

105

1.32

-1.30

Unit C

-0.85

57

0.71

-0.60

Unit D

-0.79

27

0.34

-0.27

Unit E

-0.76

21

0.26

-0.20

Installation X Alcohol Impairing Memory
Factor Score

3

-0.87

1.00

-0.94

Specifically, we weight each of an installation’s factor scores by the number of respondents per factor per unit. As such, unit
weights could vary slightly for different factors if slightly greater or fewer respondents in a unit completed the items comprising
each factor.

Installation Evaluation Methodology
Step 3: After computing scores for each of the factors across all the installations, we compute percentile
scores by comparing an installation’s score on a given factor to the factor scores of all other installations.
We standardize installation scores before averaging across factors because the DEOCS factors have
very different factor score distributions. For example, only 2% report (SD = 3%) “frequent memory loss
due to alcohol” whereas 84% report a “presence of workplace hostility” (SD = 10%). Thus, converting to
percentiles ensures that no risk or protective factor disproportionally contributes to the protective and risk
composite measures.
Installation X (Example): There are 744 installations with Cohesion factor score data. Of this total, there
are 484 installations with Cohesion factor scores less than Installation X’s score of 0.72, and 259
installations with Cohesion factor scores greater than 0.72. Thus, Installation X ranks in the 65th percentile
on Cohesion. Similarly, for Alcohol Impairing Memory, there are 142 installations with factor scores less
than Installation X’s score of -0.94, and 601 installations with Alcohol Impairing Memory factor scores
greater than -0.94. We repeat this ranking calculation for all protective and risk factors, so that each
installation has a percentile score on each factor (see Table 5).
Step 4: Finally, we compute a Protective Percentile Score for each installation by calculating the average
score (equally weighted) across the 11 protective factors percentiles. Similarly, we compute a Risk
Percentile Score for each installation by calculating the average score across the 11 risk factors
percentiles. Thus, both Protective and Risk Percentile Scores can range from 0 to 100.
Installation X (Example): As shown in Table 4, Installation X’s 11 Protective percentiles scores are
averaged to create the Protective Percentile Score of 61. Likewise, Installation X’s 11 Risk percentiles
scores are averaged to create the Risk Percentile Score of 39.

Table 5: Converting from Factor Scores to Protective and Risk Percentile Scores for Installation X
(Example)
Total Number of
Installations

Installation X
Factor Score

Installation X
Percentile Score

Cohesion

744

0.72

65

Connectedness

744

0.73

65

Engagement & Commitment

744

0.71

72

Fairness

744

0.44

52

Inclusion

744

0.61

52

Morale

744

0.38

64

Safe Storage for Lethal Means

744

0.26

81

Work-life Balance

744

0.73

85

Leadership Support (Immediate Supervisor)

744

0.75

47

Transformational Leadership (Commander)

744

0.68

43

Transformational Leadership (Senior NCO)

738

0.67

42

Protective Factors

61

Protective Percentile Score
Risk Factors
Alcohol Impairing Memory

744

-0.94

19

Binge Drinking

744

-0.56

32

Stress

744

-0.37

43

Passive Leadership (Commander)

744

-0.66

49

Passive Leadership (Senior NCO)

738

-0.68

49

Installation Evaluation Methodology
Toxic Leadership (Immediate Supervisor)

744

-0.73

27

Toxic Leadership (Senior NCO)

738

-0.65

32

Racially Harassing Behaviors

744

-0.64

44

Sexist Behaviors

744

-0.89

26

Sexually Harassing Behaviors

744

-0.44

58

Workplace Hostility

744

0.68

61

Risk Percentile Score

39

Note: Because not all units contain senior non-commissioned officers (NCO), these factors on the DEOCS were omitted for some
installations.

Further Analysis. Once installations have been identified according to their Protective and Risk
Percentile Scores, a more granular evaluation approach can be undertaken. This includes 1) examining
individual factors comprising the percentiles to determine whether some installations score especially low
or high on a few protective or risk factors; 2) considering the distribution of Protective and Risk Percentile
Scores across units to determine the potential influence of unit-level microclimates; and 3) analyzing
demographic differences (male vs. female, non-Hispanic White vs. minority, and enlisted vs. officer) on
select factors.

Suppression Rules
To protect the anonymity of survey respondents, data from units with fewer than 16 total respondents are
not included in this analysis. In addition, units with fewer than 5 respondents for any factor are
suppressed from data visualization. However, data suppressed at the unit-level are included in the
calculation of installation-level Protective and Risk Percentile Scores by combining these results with the
results of other units at the same installation. This level of aggregation addresses concerns regarding
small sample size and therefore any concerns regarding anonymity.

Installation Evaluation Methodology
Appendix: Background on DEOCS 5.0
The redesigned DEOCS (Defense Organizational Climate Survey) 5.0 assesses 19 protective and risk
factors that can impact a unit/organization’s climate and ability to achieve its mission.
Protective Factors are attitudes, beliefs, and behaviors associated with positive outcomes for
organizations or units. Higher favorable scores on protective factors are linked to a higher likelihood of
positive outcomes, such as improved performance or readiness and higher retention and are also linked
to a lower likelihood of negative outcomes, such as suicide, sexual harassment, and sexual assault. The
DEOCS 5.0 identifies 10 Protective Factors. However, for the purposes of this analysis, transformational
leadership ratings for the unit/organization leader and the Senior NCO, if applicable, are treated as two
separate factors.













Cohesion assesses whether individuals in a workplace care about each other, share the same
goals, and work together effectively. Cohesive organizations are linked to improved readiness
and retention, and a lower likelihood of sexual assault, sexual harassment, and suicide.
Connectedness measures perceptions of closeness to a group and satisfaction with one’s
relationship to others in the group. Higher connectedness is linked to a lower likelihood of suicidal
ideation.
Engagement & Commitment measures one’s vigor, dedication, and absorption in work and
commitment to the job and organization. Higher levels of engagement and commitment are linked
to higher levels of readiness, performance, and retention, and a lower likelihood of suicide.
Fairness is the perception that organizational policies, practices, and procedures, both formal
and informal, regarding information sharing, job opportunities, promotions, and discipline are
based on merit, inclusion, equality, and respect. Fair organizations are linked to higher retention
and lower levels of racial and ethnic discrimination.
Inclusion indicates whether organization members feel valued and respected by their peers and
leadership, and if they feel involved in decision-making and information-sharing. Inclusive
organizations are linked to lower rates of discrimination and higher readiness.
Morale measures whether organizations or units complete tasks with enthusiasm and confidence
in the mission. Organizations with high morale are linked to improved readiness, higher retention,
and a lower likelihood of sexual assault.
Safe Storage for Lethal Means measures how often one keeps objects that can be used to hurt
themselves or others, such as firearms and medication, safely stored in their living space.
Keeping lethal means safely stored more often is linked to a lower likelihood of suicide.
Work-Life Balance measures one’s perception that the demands of their work and personal life
are compatible. A work-life balance is linked to higher retention, improved readiness, and a lower
likelihood of suicidal ideation.
Leadership Support is the perception of support for individual goals (including career goals),
perceptions about leadership communication, and trust in leadership. Respondents rate their
immediate supervisor on this factor. Organizations with supportive leaders are linked to improved
readiness, higher retention, and a lower likelihood of suicidal ideation.
Transformational Leadership is a leadership style that inspires staff by providing motivation and
meaning to their work, giving attention to individuals’ unique needs, and directing their focus to
higher goals, such as those of the mission. Respondents rate their unit/organization leader and
their Senior NCO, if applicable, on this factor. Organizations with transformational leaders are
linked to improved job performance, job satisfaction scores, and leadership satisfaction scores.

Risk Factors are attitudes, beliefs, and behaviors associated with negative outcomes for organizations or
units. Higher unfavorable scores on risk factors are linked to a higher likelihood of negative outcomes,
such as suicide, sexual harassment, and sexual assault and are also linked to a lower likelihood of
positive outcomes, such as higher performance, readiness, and retention. The DEOCS 5.0 identifies 9
Risk Factors. However, for the purposes of this analysis, passive leadership ratings and toxic leadership
ratings for the unit/organization leader and the Senior NCO, if applicable, were treated as separate
factors.

Installation Evaluation Methodology















Alcohol Impairing Memory measures how often, during the last 12 months, one was unable to
remember what happened the night before due to drinking alcohol. Frequent memory loss due to
alcohol is linked to a higher likelihood of sexual assault and suicide.
Binge Drinking measures how often one consumes four or more drinks (for females) and five or
more drinks (for males) on one occasion. Frequent binge drinking is linked to a higher likelihood
of sexual assault and suicide.
Stress measures the feeling of emotional strain or pressure. Higher levels of stress are linked to
higher likelihood of suicide and suicidal ideation, and lower levels of readiness and retention.
Passive Leadership is a leadership style that avoids and neglects mistakes or problems until
they can no longer be ignored. Respondents rate their unit/organization leader and their Senior
NCO, if applicable, on this factor. Organizations with passive leaders are linked to lower levels of
readiness and a higher likelihood of sexual harassment.
Toxic Leadership behaviors include disregard for subordinate input, defiance of logic or
predictability, and self-promoting tendencies. Respondents rate their immediate supervisor and
their Senior NCO, if applicable, on this factor. Organizations with toxic leaders are linked to lower
organizational commitment, lower retention, and fewer respectful behaviors between members.
Racially Harassing Behaviors describe unwelcome or offensive experiences of organization
members based on their race or ethnicity. The presence of racially harassing behaviors in
organizations is linked to higher rates of legally-defined racial/ethnic harassment as well as lower
levels of readiness and retention.
Sexually Harassing Behaviors assesses the presence of unwelcome sexual advances,
requests for sexual favors, and offensive comments or gestures of a sexual nature. The presence
of sexually harassing behaviors in organizations is linked to a higher likelihood of legally-defined
sexual harassment (in which the behaviors are sufficiently persistent and severe), gender
discrimination, sexual assault, suicide, and lower levels of readiness.
Sexist Behaviors describe situations where someone is mistreated or excluded based on their
sex or gender. The presence of sexist behaviors in organizations is linked to higher rates of
legally-defined gender discrimination (in which the experiences harmed or limited their career)
and sexual assault, as well as lower levels of readiness.
Workplace Hostility measures the presence of aggressive behaviors directed at another
individual while at work. This aggression includes physical intimidation, verbal intimidation,
spreading rumors or negative comments about a person to undermine their status, and persistent
criticism of work or effort. Organizations with workplace hostility are linked to lower performance,
lower levels of readiness, and a higher likelihood of sexual harassment, sexual assault, and
racial/ethnic discrimination.

For more information on the DEOCS 5.0, see https://www.defenseculture.mil/Assessment-toSolutions/A2S-Home/