Installation Evaluation Metrics

PREVENTION EVALUATION AT
HIGH- AND LOW-RISK INSTALLATIONS:
DATA COLLECTION OVERVIEW

Introduce RAND’s Project
Provide an Overview of:
• How the Metrics Were Developed

ROADMAP FOR
TODAY’S
TRAINING

• How the Metrics Will Be Scored

Provide an Overview of:
• Process for Collecting Metrics
• Process for Generating Final
Assessment Using Metrics

Describe Next Steps

PROJECT OBJECTIVES

To establish a process and metrics to conduct independent
evaluations of installations with unhealthy and healthy climate.

To help DoD’s Under Secretary for Personnel and
Readiness, Office of Force Resiliency determine the
gaps within the prevention infrastructure and other
areas of climate at DoD sites that have healthy and
unhealthy command climate.

New information not
obtainable through
other methods

INDICATORS OF
PROJECT SUCCESS

Yield data that is
actionable

A refined process and
metrics for enduring
high-risk installation
evaluations

MOVING BEYOND
COMPLIANCE WITH
INTEGRATED VIOLENCE
PREVENTION (IVP)
POLICY
FOCUS OF PROJECT

Step 1

Compliance: Is the
requirement met?

Step 2

Quality: Is it implemented in
alignment with best practice?

Step 3

Effectiveness: Does it reduce
or stop violence?

KEY METHODS TO MEET
PROJECT OBJECTIVES

OBJECTIVE I.
DEVELOP
P RO C E S S A N D
METRICS FOR
E VA L UAT I N G
SITES

OBJECTIVE II.
IDENTIFY GAPS IN
PREVENTION
INFRASTRUCTURE
AT SITES

• Use DoD’s Compliance
Framework
• Gap analysis comparing existing
compliance tools with the DoD
Policy on Integrated Primary
Prevention of Self-Directed
Harm and Prohibited Abuse or
Harm (DoDI 6400.09)
• Literature review
• Expert consultation

• Site visits to collect metrics
using four methods:
1. Onsite discussions
2. Document review
(collected before and
while onsite)
3. Competency survey of
leaders and prevention
personnel
4. Integrated prevention
tabletop exercise

DOD’S COMPLIANCE FRAMEWORK
GUIDES THE PROJECT
Assessed
through OSD
policy and
oversight

Assessed
through
installation
assessment

Visibility

• Robust policy
• Accurate integrated data
• Visibility of risk and compliance

Priority

• Leadership prioritization of risk
mitigation

Preparation

Implementation

• Equipped and empowered
leaders and prevention personnel

• Effective implementation of
counter measures

Improvement
Findings used to
improve policy and
data

Core metrics across all
sites

Metrics
OBJECTIVE I.
DEVELOP
PROCESS AND
METRICS FOR
EVALUATING
SITES

Experimental metrics
targeting specific
prevention efforts
On-site discussions

Document review

Process
Competency survey

Scenario-based exercise

DOD COMPLIANCE FRAMEWORK + GAP ANALYSIS =
NINE CORE AREAS FOR SITE EVALUATION
PRIORITY:

PREPARATION:

IMPLEMENTATION:

Does Leadership consistently emphasize as a
priority….

Are Leaders and Prevention personnel
equipped and empowered (with a clear line
of site across the chain of command) to …..

Are efforts implemented with quality
and seamlessly integrated…

Protective
Environment

1

2

3

Integrated Primary
Prevention

4

5

6

Stakeholder
Engagement

7

8

9

CORE

• Priority: Higher-level leadership sets the tone AND
sustains consistent focus on harmful behaviors.
• Preparation: Prevention personnel and intermediate
leadership are equipped with the ability, and exist
within a structure, that incentivizes and supports
addressing harmful behaviors.
• Implementation: Approach aligns with best practice
and done well (i.e., with high quality).

CORE DEFINITIONS

• Healthy and Protective Environment: Command
climates can positively or negatively impact behaviors
such as sexual assault and harassment.
• Integrated Prevention: Effective prevention targets
a mix of risk and protective factors that are both
common across problem areas as well as unique
aspects.
• Stakeholder Engagement: Outcomes can be
improved when multiple stakeholders have genuine
involvement in prevention activities.

Metrics come in many different forms

DEFINITION OF
METRIC

Data representative of an
organization’s actions, abilities, and
overall quality

A set of organizational criteria that can be used
for self-assessment to determine the extent to
which an installation’s sexual assault prevention
efforts align with best practices

An individual survey assessing practitioner
competencies (skills, knowledge, etc.) for
conducting sexual assault prevention

A checklist that asks whether certain policies
are being enforced (yes or no)

Key Assumptions
• Must be tied to one of the nine areas (e.g., leadership
priority for protective environment)

FINDING THE
RIGHT METRICS

• Must reflect a concept that is either relevant across an
installation (not specific to a single office or problem
behavior) or would be insightful about a specific high-risk
unit

Selection of Final Metrics
• Will be done with parsimony in mind (i.e., the fewest number of
metrics that span across the nine areas for site evaluation)

• Must be feasible within the process/methods we have at our
disposal
• Must not be duplicative of the other data that’s already being
collected

Introduce RAND’s Project
Provide an Overview of:
• How the Metrics Were Developed

ROADMAP FOR
TODAY’S
TRAINING

• How the Metrics Will Be Scored

Provide an Overview of:
• Process for Collecting Metrics
• Process for Generating Final
Assessment Using Metrics

Describe Next Steps

RAND GENERATED AN INITIAL LIST OF
EVIDENCE-INFORMED SUBDIMENSIONS
REFLECTING THE NINE DIMENSIONS
PRIORITY:

PREPARATION:

IMPLEMENTATION:

Does Leadership consistently emphasize as a
priority….

Are Leaders and Prevention personnel
equipped and empowered (with a clear line
of site across the chain of command) to …..

Are efforts implemented with quality
and seamlessly integrated…

CORE
Protective
Environment

Integrated Primary
Prevention

Stakeholder
Engagement

Subdimensions

1

4

7

•

2
1.1. Leaders consistently emphasize
the importance
of a healthy protective environment

•

1.2. Leaders consistently deter negative behaviors

•

1.3. Leaders hold subordinates
5 accountable for timely
action

•

1.4. Leaders reinforce positive behaviors

•

1.5. Leaders role models positive behaviors

8

3

6

9

EXPERT PANEL HELPED DEVELOP AND
REFINE FINAL SET OF SUBDIMENSIONS

Experts Names and Affiliation*

• Dr. Eric R. Pedersen, Senior behavioral scientist at the RAND
Corporation
• Christine Gidycz, developer of The Ohio University Sexual Assault
Risk Reduction Program
• Jacquelyn W. White, Emerita Professor of Psychology and former
director of Women’s and Gender Studies at the University of North
Carolina at Greensboro, where she also served as Associate Dean
for Research in the College of Arts and Sciences
• Andra Teten Tharp, Senior Prevention Advisor in the US Department
of Defense Sexual Assault Prevention and Response Office
• Mark Greenberg, Emeritus Professor, Human Development and
Family Studies, Founding Director, Edna Bennett Pierce Prevention
Research Center, Penn State University
*Affiliation at the time they provided feedback.

Used RAND Appropriateness Method to derive
final set of metrics.
Consensus ratings from experts on the
importance and validity of the metrics.
•

•

Validity was defined as having adequate scientific evidence
or professional consensus exists to support a link between
the metric and the effectiveness of efforts to prevent selfdirected harm and abusive and harmful behaviors
Importance was defined as adherence to the
subdimension is a primary driver of the effectiveness of
efforts to prevent self-directed harm and abusive and
harmful behaviors.

THEN RAND DEVELOPED AN APPROACH TO
GENERATE SCORES FOR EACH DIMENSION
Dimension:
One of the nine
dimensions framing
the assessment

MATURITY SCORE ranging from
0 to 5 reflecting the extent to which
this dimension was evident at the
installation

1-Protective Environment – Priority

Scoring guidance details how the
sufficiency of subdimensions leads to
the score for each dimension

Subdimension:
Key concepts that
define each
dimension

1.1.Leaders consistently deter negative
behaviors

Scores of sufficient or insufficient for
each the subdimensions based on
presence of data elements

Levels of Data
Collection

Data element:
A set of descriptive
indicators for each
subdimension

1.1.1. Leaders monitor
progress on relevant metrics
of climate
Example

Scoring guidance details
how the presence of data
elements leads to a score
of sufficient or insufficient
for each subdimension
Assessed based on presence of
evidence that consistently
supports each data element

Getting to the
Maturity Score

MATURITY
SCORING SYSTEM
TRACKS PROGRESS

• A set of characteristics, attributes, indicators, or
patterns that represent progression and achievement
in a particular domain or discipline
• Evaluate against benchmarks of best practices and may
incorporate standards or other codes of practice that
are important in a particular domain or discipline.

• Architecturally, maturity models typically have “levels”
along an evolutionary scale that defines measurable
transitions from one level to another.

EXAMPLE OF HOW TO GENERATE ONE OF
THE NINE MATURITY SCORES
PRIORITY:

PREPARATION:

IMPLEMENTATION:

Does Leadership consistently emphasize as a
priority….

Are Leaders and Prevention personnel
equipped and empowered (with a clear line
of site across the chain of command) to …..

Are efforts implemented with quality
and seamlessly integrated…

CORE
Protective
Environment

Integrated Primary
Prevention

Stakeholder
Engagement

Subdimensions

1

4

7

• 1.1. Leaders consistently emphasize
the importance of a healthy 2
protective environment

• 1.2. Leaders consistently deter
negative behaviors

3

5

6

• 1.4. Leaders reinforce positive
behaviors
8

9

• 1.3. Leaders hold subordinates
accountable for timely action

• 1.5. Leaders model positive
behaviors

P ROT E C T I V E E N V I RO N M E N T: P R I O R I T Y

Protective Environment—Priority
has 5 subdimensions.
The Maturity Score is based on
the sufficient presence of
these subdimensions:

5. Sufficient in all 5 subdimensions and
consistent evidence that sufficiency has been
maintained over time despite competing
priorities

4. Sufficient in all 5 subdimensions
3. Sufficient in 4 out of 5 subdimensions
2. Sufficient in 3 out 5 subdimensions
1. Sufficient in 1 or 2 out of 5 subdimensions
0. No subdimensions are sufficient

EXAMPLE OF ONE OF THE NINE CELLS
PRIORITY:

PREPARATION:

IMPLEMENTATION:

Does Leadership consistently emphasize as a
priority….

Are Leaders and Prevention personnel
equipped and empowered (with a clear line
of site across the chain of command) to …..

Are efforts implemented with quality
and seamlessly integrated…

CORE
Protective
Environment

Integrated Primary
Prevention

Stakeholder
Engagement

Subdimensions

1

4

7

•

1.1. Leaders consistently
2 of
emphasize the importance
a healthy protective
environment

•

1.2. Leaders consistently deter
5
negative behaviors

•

1.3. Leaders hold subordinates
accountable for timely action

•

1.4. Leaders reinforce positive8
behaviors

•

1.5. Leaders role models positive
behaviors

❑

❑

❑

❑

Data Elements for 1.1.
Installation leadership has an intentional and
visible vision regarding addressing
3 negative
or unwanted behaviors (e.g., sexual
assault/harassment, alcohol use, suicide)
Communications from leaders include
efforts to address potential stigma (e.g.,
normalizing of experiences that might lead
6
to problem behaviors or disclosure of such
behaviors)
Leadership voice support of primary
prevention activities such as education and
training activities or information awareness
9
campaigns
Leaders have, follow, and widely share a
strategic prevention plan AND revisits this
statement/plan regularly

P ROT E C T I V E E N V I RO N M E N T: P R I O R I T Y

Subdimension

Sufficient presence of these
subdimensions is based on a rating of
whether a set of between 1-5 data
elements is absent or present at a site.

1.1
1.2

1.3
The definition of sufficient varies by subdimension
and is based on the number of data elements.

1.4
1.5

Number of data elements

EXAMPLE OF ONE OF THE NINE CELLS

PRIORITY:

PREPARATION:

Does Leadership consistently emphasize as a
priority….

Are Leaders and Prevention personnel
Are efforts implemented with quality
equipped and empowered (with a clear
line =
and
seamlessly integrated…
Sufficient
Consistent
evidence
of site across the chain of command) to …..

supporting at least 3 out of 4

CORE
Protective
Environment

Integrated Primary
Prevention

Subdimensions

1

4

• 1.1. Leaders consistently
2
emphasize the importance of
a healthy protective
environment

7

❑

❑

• 1.2. Leaders consistently deter
5
negative behaviors
• 1.3. Leaders hold subordinates
accountable for timely action

Stakeholder
Engagement

IMPLEMENTATION:

• 1.4. Leaders reinforce positive 8
behaviors
• 1.5. Leaders model positive
behaviors

❑

❑

Data Elements for 1.1
Installation leadership has an intentional and
visible vision regarding addressing
3 negative
or unwanted behaviors (e.g., sexual
assault/harassment, alcohol use, suicide)
Communications from leaders include
efforts to address potential stigma (e.g.,
normalizing of experiences 6
that might lead
to problem behaviors or disclosure of such
behaviors)
Leadership voice support of primary
prevention activities such as education and
training activities or information awareness
9
campaigns
Leaders have, follow, and widely share a
strategic prevention plan AND revisits this
statement/plan regularly

EXAMPLE OF ONE OF THE NINE CELLS

PRIORITY:

PREPARATION:

IMPLEMENTATION:

Does Leadership consistently emphasize as aSubdimensions
Are Leaders and Prevention personnel
Are efforts implemented with quality
priority….
and empowered (with a clear line
and seamlessly integrated…
• 1.1. Leadersequipped
consistently
emphasize
Sufficient
of site across the chain of command)
to ….. = Consistent evidence

the importance of a healthy
protective environment

CORE
Protective
Environment

1

supporting both

• 1.2. Leaders consistently deter
2
negative behaviors

❑

• 1.3. Leaders hold subordinates
accountable for timely action
Integrated Primary
Prevention

Stakeholder
Engagement

4

7

• 1.4. Leaders reinforce positive
behaviors
• 1.5. Leaders model positive
behaviors

5

8

❑

Data Elements for 1.2
Leaders can identify and enforce
the specific
3
policies governing violations and negative
behaviors (e.g., as identified in the DoDI DoDIs 1350.02, 1438.06, 1010.04, 1020,03,
1020.04, 6490.16, 6495.02, DODD 1020.02E
and 1440.1)
Leaders monitor progress6on relevant metrics
of climate (e.g., sick call, injuries, disciplinary
action, attrition, suicide rates, referrals to
FAP), including measures related to service
members, DoD civilians, military families, and
9
other personnel

EXAMPLE OF ONE OF THE NINE CELLS
Subdimensions
• 1.1. LeadersPREPARATION:
consistently emphasize
the importance
of a healthy
Does Leadership consistently emphasize
as a Are Leaders
and Prevention personnel
priority….
and empowered (with a clear line
protective equipped
environment
PRIORITY:

IMPLEMENTATION:
Are efforts implemented with quality
and seamlessly integrated…

of site across the chain of command) to …..

Protective
Environment

Sufficient = Consistent evidence
supporting both

• 1.2. Leaders consistently deter
negative behaviors

CORE

1

Integrated Primary
Prevention

4

Stakeholder
Engagement

7

2
• 1.3. Leaders hold subordinates
accountable for timely action
• 1.4. Leaders reinforce positive
behaviors
• 1.5. Leaders model positive
behaviors

5

8

Data Elements for 1.3
❑ Leadership holds subordinates
3 responsible for
ensuring timely discipline measures are taken
for service members that perpetrate reported
cases (e.g., in case of harassment, assault,
domestic abuse)
❑ Leadership holds subordinates
6 responsible for
referring service members to needed
treatment (e.g., for substance use, suicide) in a
timely way when an issue has been identified

9

Subdimensions
EXAMPLE OF ONE
OF THE NINE CELLS
• 1.1. Leaders consistently emphasize
the importance of a healthy
protective environment
• 1.2. Leaders consistently deter
PREPARATION:
negative behaviors

PRIORITY:

Does Leadership consistently emphasize as a
priority….

CORE
Protective
Environment

1

Integrated Primary
Prevention

4

Stakeholder
Engagement

7

Sufficient =IMPLEMENTATION:
Consistent evidence
supportingAre
each
Are Leaders and Prevention personnel
efforts implemented with quality
equipped and empowered (with a clear line

and seamlessly integrated…

• 1.3. Leaders hold subordinates
Data Elements for 1.4 and 1.5
of site across the chain of command) to …..
accountable for timely action
❑ Leadership rewards or recognizes appropriate
behavior that support positive norms in a
• 1.4. Leaders reinforce positive
timely manner (e.g., bystander behaviors,
behaviors
proper handling of harassment/assault
reports;
2
3
demonstrating strong diversity and inclusion
• 1.5. Leaders model positive
behaviors and principles)
behaviors
❑ Leadership are observed modeling appropriate
behaviors, such as addressing problematic
behaviors or demonstrating a commitment to
5
6
diversity and inclusion

8

9

THREE “EXPERIMENTAL” METRICS FOCUS ON
IMPLEMENTATION IN SPECIFIC AREAS

IMPLEMENTATION
•

10. High risk groups

11. Lethal means

12. Responsible alcohol
use

All three experimental metrics include
Subdimensions assessing
• whether the approach being
implemented is:
1. Comprehensive
2. Based on the latest
research/evidence about what
works
3. Evaluated
4. Continuously improved
5. Whether service members
engagement and resistance to
the approach is monitored and
addressed

Lethal Means
Subdimensions

11.1
11.2
11.3

11.4
11.5

Number of data elements

THREE “EXPERIMENTAL” METRICS FOCUS ON
IMPLEMENTATION IN SPECIFIC AREAS
IMPLEMENTATION
•

10. High risk groups

11. Lethal means

12. Responsible alcohol
use
•

All three experimental metrics include
Subdimensions assessing
• (1) whether the approach being
implemented is:
•
Comprehensive
•
Based on the latest
research/evidence about
what works
•
Evaluated
•
Continuously improved
•
(2) whether service members
engagement and resistance to
the approach is monitored and
addressed
Data elements vary by content of
specific area

Data Elements for Comprehensive Lethal
Means Approach
❑ Promotes public health messaging and education on
how to recognize risk factors and how safe
messaging can help protect or prevent suicide
through messages tailored to service members
❑ Takes into account the various types of means and
environmental conditions that contribute to deaths
by suicide (e.g., firearms, opioids, bridges)
❑ Promotes awareness about the availability of lethal
means safety programs (e.g., safe gun storage)
❑ Includes necessary onsite partners (e.g., public
affairs, mental health, health clinics and hospitals)
and offsite partners (e.g., firearm dealers, gun
owners)
❑ Incorporate suicide awareness as a basic tenet of
firearm safety and gun ownership

THREE “EXPERIMENTAL” METRICS FOCUS ON
IMPLEMENTATION IN SPECIFIC AREAS
IMPLEMENTATION
•

10. High risk groups

11. Lethal means

12. Responsible alcohol
use

•

All three experimental metrics include
Subdimensions assessing
• (1) whether the approach being
implemented is:
•
Comprehensive
•
Based on the latest
research/evidence about
what works
•
Evaluated
•
Continuously improved
•
(2) whether service members
engagement and resistance to
the approach is monitored and
addressed
Data elements vary by content of the
specific area

Data Elements for Comprehensive Lethal Means
Approach (continued…..)
❑ Ensures safety and prevention efforts targeting periods
of high suicide risk to put space and time between
service members and lethal means (e.g., guns,
prescription, over-the-counter medications, bridges,
parking garages) and provide counseling and education
❑ Uses safety technologies to reduce access to lethal
means (e.g., physical barriers, prescription lockboxes,
crisis call boxes), particularly at places which may be
considered suicide ‘hotspots’
❑ Ensures health clinics, hospitals, and mental health
facilities use standardized protocols to allow for the
early identification of suicide risk, safety planning
interventions, and lethal means safety follow-up
❑ Ensure all health providers, law enforcement officers,
and others that may be likely to come into contact
with service members at-risk for suicide are trained on
how to interact with and reduce access to lethal
means among service members who may be at risk for
suicide
❑ Allows for the safe disposal of opioids and other
prescription and over-the-counter medications

Introduce RAND’s Project
Provide an Overview of:
• How the Metrics Were Developed

ROADMAP FOR
TODAY’S
TRAINING

• How the Metrics Will Be Scored

Provide an Overview of:
• Process for Collecting Metrics
• Process for Generating Final
Assessment Using Metrics

Describe Next Steps

KEY METHODS TO MEET
PROJECT OBJECTIVES

OBJECTIVE I.
DEVELOP
P RO C E S S A N D
METRICS FOR
E VA L UAT I N G
SITES

OBJECTIVE II.
IDENTIFY GAPS IN
PREVENTION
INFRASTRUCTURE
AT SITES

• Use DoD’s Compliance
Framework
• Gap analysis comparing existing
compliance tools with the DoD
Policy on Integrated Primary
Prevention of Self-Directed
Harm and Prohibited Abuse or
Harm (DoDI 6400.09)
• Literature review
• Expert consultation

• Site visits to collect metrics
using four methods:
1. Onsite discussions
2. Document review
(collected before and
while onsite)
3. Competency survey of
leaders and prevention
personnel
4. Integrated prevention
tabletop exercise

1. ONSITE DISCUSSIONS - EXAMPLE
SCHEDULE
DAY 1 TEAM ONE
0930-1030 O1-O3 Men
1100-1200 O4-O5 Men
1200-1330 Lunch
1330-1430 O6 Men
1500-1600 MEO/EEO Staff

DAY 1 TEAM TWO
0800-0900 Installation command and command team
0930-1030 O1-O3 Women
1100-1200 O4-O5 Women
1200-1330 Lunch
(O6 women on Day 3)
1330-1600 Prevention Tabletop Exercise

DAY 2 TEAM ONE
0800-0900 E1-E4 Men (#1)
0930-1030 E1-E4 Men (#2)
1100-1200 E5-E6 Men
1200-1330 Lunch
1330-1430 E7-E9 Men
1500-1600 Community support services

DAY 2 TEAM TWO
0800-0900 E1-E4 Women (#1)
0930-1030 E1-E4 Women (#2)
1100-1200 E5-E6 Women
1200-1330 Lunch
1330-1430 E7-E9 Women
1500-1600 FRG/SFRG/FRP/ Key Spouse

DAY 3 TEAM ONE
0800-0900 Chaplains and MFLC
1030-1130 Physical health (primary care, ER)
1200-1330 Lunch
1330-1430 Sexual assault and Victim Advocates
1500-1600 IG and law enforcement

DAY 3 TEAM TWO
0800-0900 Mental health
1030-1130 Substance abuse
1200-1330 Lunch
1330-1430 Family advocacy
1500-1600 O6 Women

DAY 4 TEAM ONE
0800-0900 Farewell meeting with installation commander and command team

DAY 4 TEAM TWO
0800-0900 Any remaining key prevention personnel

1. ONSITE DISCUSSIONS – STANDARDIZED
PROTOCOLS PROVIDE DETAILED INSTRUCTIONS

2. DOCUMENT REVIEW
Flagship efforts
• For multiple harmful behaviors (problematic alcohol use,
sexual assault, etc)

• Asks for details of the effort (e.g., the evidence base,
numbers of service members exposed, number of times
run, evaluation outcomes)

RAND’S DATA C ALL
ASKS ABOUT FLAGSHIP
PREVENTION EFFORTS
AND OTHER ACTIVITIES

Additional activities
• Communication between leadership and service member
• Presence of “Integrator” groups (i.e., stakeholder groups)
• Evaluation reports from prevention activities

3. COMPETENCY
SURVEY ASSESSES
≈
INDIVIDUAL
KNOWLEDGE & SKILLS

• Completed by leaders and prevention personnel at each
site at the end of the discussion groups
• Assesses knowledge and skills needed to:
• Build a protective environment (e.g., understand how the
environment can cause or maintain harmful behaviors)
• Conduct high-quality prevention (e.g., understand the key
ingredients of evidence-based prevention for sexual
assault)
• Engage service members in reducing self-directed harm,
abusive and harmful behaviors (e.g., work with service
members in such a way that they have meaningful input
into decisions about prevention activities)

4. INTEGRATED PREVENTION TABLETOP EXERCISE TAPS
KNOWLEDGE AND SKILLS IN A ‘REAL WORLD’ MANNER

Step 1

Step 2

Step 3

Step 4

Agenda and
ground
rules

Introduce
situation

Unfolding/
breaking
news

Debriefing/
self-evaluation

2

3

TWO
OBJECTIVES

To exercise how
your installation
would work together
to address harmful
behaviors

To determine where
strengths and
weaknesses may lie
with respect to
integrated prevention
planning and capacity

ASSESSES FOUR AREAS RELATED TO
INTEGRATED PRIMARY PREVENTION
Partnerships

Engagement

Work across multiple offices and
personnel responsible for the prevention
of harmful behaviors

Improve prevention through
interactions with service members,
families, and community partners

Data Application

Communication

Learn about harmful behaviors and
approaches to address them

Prioritize a line of sight across the
chain of command

PROTECTIVE ENVIRONMENT: PRIORITY

Subdimension

• Multiple data sources
feed the scoring
≈

• The type and number of data
sources used for rating each
subdimension varies.

Number of
data elements

Methods
informing rating

1.1

• Onsite discussions
• Data call
• Command-installation self
assessment

1.2

• Onsite discussions
• Command-installation self
assessment

1.3

• Onsite discussions
• Data call
• Command-installation self
assessment
• Tabletop exercise

1.4

• Onsite discussions
• Data call
• Command-installation self
assessment

1.5

• Onsite discussions

PROCESS TO GET TO FINAL PROFILE
OF EACH SITE

Within 48
hours:
Prior to visit:
Review
compliance
metrics, DEOCs
and other
background
material from data
call

While onsite:
Clean notes up at
the end of each
day

Within 24
hours:
Compile and share
notes across site
visit team; Enter
survey data
collected

Review notes and
any documents
collected while
onsite and
independently
generate
maturity initial
ratings and
bullets describing
site strengths/
weaknesses

Within 72
hours:
Compile and share
ratings and
strengths/
weaknesses across
site visit team

Within 96
hours:

Within 120
hours:

Discuss any
discrepancies in
ratings and come
to consensus on
maturity ratings
for the site and
site strengths/
weaknesses

Submit final
maturity ratings
and strengths/
weaknesses to site
profile manager;
Profile manager
produces final
site profile

MATURITY SCORES WILL FEED INTO SIMPLE
VISUALIZATIONS FOR THE FINAL SITE PROFILE

Final format TBD, but some sample visualizations are presented next!

Introduce RAND’s Project
Provide an Overview of:
• How the Metrics Were Developed

ROADMAP FOR
TODAY’S
TRAINING

• How the Metrics Will Be Scored

Provide an Overview of:
• Process for Collecting Metrics
• Process for Generating Final
Assessment Using Metrics

Describe Next Steps

NEXT STEP IS TO PILOT
TEST & TRAIN DATA
COLLECTORS TO
RELIABILITY

Pilot Test
• Assemble DoD representatives to
review data collection methods and
visualization options
Reliability training
• Assemble data collectors
• Present ‘fake’ data

• Have data collectors make ratings
• Discuss discrepancies

TIMELINE

Upcoming
Deliverables

2020
Apr

August 31, 2021*
▪ Site profiles

2021
May

Develop
process and
metrics

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

Conduct and
summarize
evaluations of
sites (i.e., site profiles) Lessons learned memo

RAND methods
report
*Site profiles will be sent to OFR in batches as they are completed (i.e., on a rolling basis) and timing will be based on timing of completed site visit.

Jun

QUESTIONS?

Joie Acosta
Senior Behavioral Scientist
RAND Corporation
1200 S. Hayes St.
Arlington,VA 22202
703-413-1100 x5324
[email protected]

Matthew Chinman
Senior Behavioral Scientist
RAND Corporation
4570 5th Ave
Pittsburgh, PA 15213
412 683-2300 x 4287
[email protected]