0704-0610_osie_ssb_10.8.2021

NOTE: Complete Part B for Survey ICR Requests

SUPPORTING STATEMENT – PART B

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

If the collection of information employs statistical methods, the following information should be provided in this Supporting Statement:

1.  Description of the Activity

These information collections support a high-visibility requirement directed in Secretary of Defense Memorandum, “Immediate Actions to Counter Sexual Assault and Harassment and the Establishment of a 90-Day Independent Review Commission on Sexual Assault in the Military,” February 26, 2021. Immediate Action 2 directed the Under Secretary of Defense for Personnel and Readiness (USD(P&R)) to develop a plan of action and milestones to conduct high risk installation evaluations. In Memorandum “Plan of Action and Milestones for High Risk Installation Evaluations,” March 30, 2021 USD(P&R) approved the plan of action and milestones. From June 2021 – January 2021, 20 site visits served as pilot sites for Onsite-Installation Evaluations (OSIEs) in order to test standardized methods and metrics to gain greater visibility into the prevention capabilities of installations at risk for interpersonal and self-directed violence.

Beginning in 2023, DoD will evaluate 30 DoD installations on a biennial basis where the military community is at increased or decreased risk for destructive behaviors as evidenced by risk and promise scores, which will include a measure of unhealthy command climate. The biennial site visits will utilize methods and metrics refined during the initial pilot. Effective prevention decreases risk factors and increases protective factors. On-site evaluations will answer: Are the installation’s prevention capabilities poised to address the risk detected on the command climate assessment? If so, what is going right that could be replicated elsewhere? If not, what concrete actions could be taken to improve the installation’s ability to address risk and enhance promise?

DoD Office of Force Resiliency (OFR) will identify 30 DoD installations to take part in each biennial OSIE. The methods for the selection of participating sites will be included in TAB A. Units and installations of risk and promise will be identified based on analysis of risk and protective factors for prohibited behaviors, including the command climate assessments that formed the basis of pilot site identification. Sites with extreme scores for risk or promise will be selected for site visits. Identified sites for the 2023 biennial OSIE are TBD.

At each site, in addition to assessing installation assets (e.g., prevention personnel that provide services to the entire installation) data will be collected from the 1-3 units that drive the installation’s risk and protective factor scores. There will be three data sources: 1) responding to a “request for information”; 2) participating in discussions during site visits; and 3) completing a survey. The discussions and survey constitute the information collections covered in this request.

The request for information asks a series of questions about experiences conducting prevention activities targeting a variety of negative behaviors at each location, including sexual assault and harassment, substance abuse, suicide, and domestic violence. The request for information also asks what kind of prevention activities service members conduct, what their relationships are like with their colleagues at their location who also are involved in prevention, and any relevant documents about the prevention activities conducted. The request for information will be sent before the site visit.

The site visit discussions will involve staff from DoD (with support from a contract team, TBD) asking questions to follow-up on data elements in the request for information, along with additional questions about prevention at each location. The duration of site visits will vary based on the number of engagements planned for each site and the number of units selected at each installation, but are estimated to last between three and ten days. Depending upon roles of each service member at the different locations, they may be asked to participate in only one day or in multiple days of OSIE discussions.

The survey asks questions about individual levels of knowledge and skills conducting prevention activities. Depending on their role, they may be asked to participate in all three activities, two activities, or just one. Participation will be determined in consultation with contract and OFR staff. The survey will be collected on paper at the site visit. Surveys will be completed in on-site meetings and then transported back to the agency for data entry.

The request for information and site visit discussions will have identifiers, including the name of the respondent, email addresses, and phone numbers. Surveys will not have any individual identifiers. None of the information is sensitive, but mostly describes activities that while not well known, are public (existence of prevention programs at an installation). Some of the information will be about topics that are not publicly known—e.g., the individual prevention competencies of an individual respondent.

The respondents will be a cross section of personnel at the selected installations that fall into two general categories. First, are personnel that are specific to prevention and intervention activities relating to a variety of negative behaviors (e.g., alcohol use, suicide, sexual harassment). These include: Sexual assault response coordinators, victim advocates, Family Advocacy Program staff, MEO/EEO staff, Mental Health Professionals, Enlisted medical personnel (e.g., medics, corpsmen), Inspectors general and misdemeanor and felony-level law enforcement representatives, Chaplains, MWR and community/ support services staff, and Physical health professionals. We are collecting data from these individuals because they have first-hand knowledge of how prevention activities are carried out at the installation. In most cases, given the size of the identified sites, samples of participants will represent a census of the personnel at these locations.

The second category are general personnel that will be important to talk to for their perceptions of how prevention is prioritized and experienced at the installation and will include (the target sample is in parentheses): Installation commander, E1-E4 Men (10), E1-E4 Women (10), O1-O3 Men (10), O1-O3 Women (10), E5-E6 Men (10), E5-E6 Women (10), O4-O5 Men (10), O4-O5 Women (10), E7-E9 Men (10), E7-E9 Women (10), O6 Men (10), O6 Women (10), First sergeants (5 or more). These individuals will be selected from the specific units that drive command climate risk or promise at the site. We will use personnel data to determine how representative these respondents are of the total base population.

Respondents’ participation in these information collections is voluntary.

2.  Procedures for the Collection of Information

There will be three data sources: 1) responding to a “request for information”; 2) participating in discussions during a site visit; and 3) completing a survey. Data sources 1 and 2 will be similar questions. The request for information will be sent ahead via an emailed Word document to a point of contact (POC) at each installation. The POC will arrange to have the relevant individual(s) answer the questions in the Word document and send back to the government representative leading the OSIE team by email. In our experience, these types of requests for information are often returned incomplete, unclear, or inaccurate. Thus, at the site visits, site visit staff will follow up to confirm the information and fill in any missing data. During the site visit, various groups of individuals specified above will be organized into a series of discussion groups in one to one and a half hour slots (see Table 1). Although general guidance will be provided, each installation will determine the best individuals to participate.

Table 1. Hypothetical schedule of data collection at each installation
	Tuesday		Wednesday		Thursday		Friday
	Team 1	Team 2	Team 1	Team 2	Team 1	Team 2	Team 1	Team 2
0900-1000	Installation commander/ command team in-brief and interviews	Tour of installation housing and work areas	E1-E4 Men (10)1	E1-E4 Women (10)	MWR and community/ support services staff (10)	Chaplains (5 or more)	Farewell meeting with installation commander, if s/he would like	Any other type of prevention staff (e.g., AF Violence Prevention Integrators)
1030-1130	O1-O3 Men (10)	O1-O3 Women (10)	E5-E6 Men (10)	E5-E6 Women (10)	FRG/SFRG/FRP/ Key Spouse staff (10)	Sexual assault response coordinators, victim advocates, Family Advocacy Program staff (10)
1130-1300 Lunch on the installation
1300-1400	O4-O5 Men (10)	O4-O5 Women (10)	E7-E9 Men (10)	E7-E9 Women (10)	MEO/EEO staff (5 or more)	Mental Health Professionals* (10)
1430-1530	O6 Men (10)	O6 Women (10)	First sergeants (Army/AF/MC, any equiv. Navy would like to provide) (5 or more)	Enlisted medical personnel (e.g., medics, corpsmen) (10)	Inspectors general AND both misdemeanor and felony-level law enforcement representatives (10)	Physical health professionals** (10)
1 Parentheses indicate project sample size *Mental health professional group should include at least two of each of the following: (a) psychiatrists, (b) psychologists or social workers, and (c) MFLC counselors (Military and Family Life Counselors) AND at least one staff member from the substance/drug and alcohol abuse program **Physical health professionals should include at least two each of the following: (a) primary care physicians (b) emergency room nurses (c) physician assistants FRG = Family Readiness Group (Navy), SFRG = Soldier and Family Readiness Group (Army), FRP = Family Readiness Program (Marine Corps), Key Spouse program is Air Force; MWR = morale, welfare, recreation programs; MEO = Military Equal Opportunity program for military personnel, EEO = Equal Employment Opportunity program for civilian personnel

The questions for 1 and 2 will be open-ended questions about the prevention activities. The discussions will NOT be recorded, but a notetaker will be part of the site visit team and will take detailed notes. Those notes will be hand carried back to the agency to be organized and used for analysis. Data source 3 will be a survey of individual competencies to carry out prevention activities. The individuals slated for this survey are those personnel that are specific to prevention and leaders who could possibly oversee those individuals. This could be a mix of enlisted and officers depending how prevention tasks are assigned at a site. These individuals will complete the survey at their respective discussion group slot during the site visit. The survey will be paper and pencil and collected by site visit staff who will hand carry the completed forms back to the agency for data entry and analysis.

3.  Maximization of Response Rates, Non-response, and Reliability

High response rates are expected given this data collection has been ordered by the Secretary of Defense. The request for information (RFI) was the most challenging to have completed during the pilot OSIEs. In order to ease the administrative burden and increase response rate and time, the RFI has been streamlined significantly. Regardless of RFI success, we specifically are employing the site visit mechanism to ensure that all questions will have complete responses. Regarding the survey, we will administer it at the end of each discussion group time slot and immediately collect the surveys back. Thus, we expect the response rates will be very high.

We will use multiple procedures to ensure high reliability. Each site visit will comprise three or four government individuals (team lead and policy office representatives), two contract staff will lead the discussion (accompanied by a notetaker). There will be four to eight individuals on each OSIE with a total project team projected to comprise seven to twelve total personnel. All staff (DoD and contractors) will be trained to use the site visit protocol using hypothetical data. For data analysis teams, training will continue until at least an interrater reliability score of .8 is achieved across all raters. After each visit, the raters from each site visit team will independently make their ratings. Inter-rater reliability will be calculated using these separate scores. Then, the raters will discuss their ratings and come to a census when they disagree. It is this consensus score that will be used in analysis.

The data collection is not intended to be generalizable to a larger universe. That is because the data collection is intended to be an installation-level evaluation of that installation’s activities. The data will be used to help each installation improve as well as provide an overview of progress to DoD leadership.

4.  Tests of Procedures

Prior to site visits for the pilot OSIEs, RAND convened an expert panel of five subject matter experts who reviewed and rated measures for validity and importance. Also, RAND administered the discussion questions to other DoD personnel who were not involved with the project otherwise as a further check on wording and feasibility (the data was not retained). In addition, RAND asked each respondent at the end of each group discussion slot about their perceptions of the process (e.g., what worked well; what should be improved). Feedback from group discussion during pilot OSIEs were used to refine the data collection for future use.

5.  Statistical Consultation and Information Analysis

a. Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

Matthew Chinman, Ph.D., RAND Corporation, 4570 5th Ave, Pittsburgh, PA 15213, 412 683-2300 x 4287

b. Provide name and organization of person(s) who will actually collect and analyze the collected information.

Contract support for biennial OSIEs is TBD. Names of DoD personnel who supported the pilot OSIEs and are intended to serve future biennial OSIEs are listed below.

Team Members	Organization/Role
Dr. Andra Tharp	DoD/Team Lead
Mr. Travis Bartholomew	DoD/Team Lead
Col Tony Haught	DoD/Team Lead
Lt Col Thomas Maguire	DoD/Team Lead
Dr. Suzanne Holroyd	DoD/Team Lead
Dr. Beverly Fortson	DoD/Analyst
Dr. Tracy Hipp	DoD/Analyst
LCDR Evette Pinder	DoD/Analyst
LtCol Ric Cloninger	DoD/Analyst
LCDR Rich Yates	DoD/Analyst
LTC Gennelle Conway	DoD/Analyst

4