SUPPORTING STATEMENT - PART A
Wingman Intervention Training (WIT) Program Evaluation Survey – 0704-XXXX
1. Need for the Information Collection
NORC has been contracted by the Department of Defense Sexual Assault Prevention and Response Office (DoD SAPRO) to conduct training/program evaluations.
As part of its contract, NORC, in collaboration with the Department of the Air Force (DAF), is conducting a baseline survey and one follow-up survey with DAF First-Term Airmen/Guardians, fidelity feedback forms from a sample of those Airmen/Guardians receiving the Wingman Intervention Training (WIT), and fidelity forms completed by the WIT implementers. These data are being collected for the purpose of a program evaluation of WIT. The survey will assess whether the intended outcomes of reducing sexual harassment/assault were achieved, and the fidelity forms will be a check on whether all the key WIT curriculum components were implemented correctly.
This study responds to the problem of sexual harassment/assault in the military and the urgent need for a rigorous evaluation of an evidence-informed Air Force program to prevent sexual harassment/assault. WIT is already being implemented at approximately 20-25 Air Force bases globally. Without a WIT program evaluation for the Air Force, the Air Force will not know if WIT should be continued, modified, or even stopped because it is having no or harmful effects.
2. Use of the Information
The purpose of the overall evaluation is to determine the effectiveness of the Wingman Intervention Training (WIT) program in preventing sexual harassment (SH) and sexual assault (SA). The current study is consistent with a call from the 2021 Independent Review Commission (IRC) on Sexual Assault in the Military report (see Hard Truths and the Duty to Change) and the Secretary of Defense’s reply to the report. The IRC made recommendations related to: Accountability; prevention; climate and culture; and victim care and support. In particular, the IRC noted the lack of high-quality program evaluation data on sexual harassment and sexual assault interventions. The IRC stated that prevention activities in military communities is greatly hampered by this lack of quality program evaluation data. The current study will help address this gap by collecting of up-to-date sexual assault and sexual harassment data from Airmen/Guardians exposed to the WIT program in comparison to Airmen/Guardians who do not receive the WIT program (through anonymous survey protocols approved by the project IRB). Conducting this survey and the associated program evaluation aligns with the IRC’s call for greater accountability in the military services in the greatly understudied area of sexual assault prevention.
Respondents are Airmen/Guardians, and will be asked to complete two surveys timed to be administered before (at baseline) and after (at follow-up) WIT training for Airmen/Guardians receiving the training, and at corresponding timepoints for the comparison sample. Recruited respondents will be First-Term Airmen/Guardians. Respondents will be administered the web-based baseline survey starting in March 2022 on a rolling basis (based on each Airmen/Guardians entry date to their First-Term Airmen/Guardians Center and thus scheduled WIT training), with a six-month intake period until September 2022. A 6-month follow-up survey starting in September 2022 will follow, the data collection for which will close March 2023. The content for the surveys will focus on SH and SA because DAF is interested in learning whether the WIT programming is effective at preventing SH and SA events and promoting active bystander behaviors to prevent SH and SA. Respondents that participate in the WIT training will also be asked to complete program fidelity feedback forms as well as the WIT intervention implementers (see below discussion).
Survey data collection overall is anonymous. At both the baseline and the 6-month follow-up survey administrations, Airmen/Guardians who consent to participate will be advanced to a secure survey webpage to answer four questions that are durable personal information (i.e., not personally identifying, but memorable to facilitate exact replication at the follow-up survey, creating components of a “self-generated unique ID” [SGID]). NORC will program an algorithm (known to a limited subset of NORC programmers on this project who will not be involved in analyzing the survey data) to scramble the 10 letters/digits of the response data to create a unique 10-digit SGID. Thus, respondents will not know their own SGID; they will not need to remember an SGID; and they will not be able to risk a breach of respondent identity by having it written down somewhere for someone to find. However, by answering the same way to the same four questions at the follow-up survey, the same SGIDs will be generated. These SGIDs will link the baseline and follow-up responses while maintaining the anonymity of participants. We are following industry best practices and believe this is the strongest SGID protocol to date to protect individual anonymity in longitudinal research, both in terms of length of the ID and the additional algorithm that will be secured separately from the survey response data set. Further, the demographic questions asked on the survey have been carefully curated to minimize the risk of deductive disclosures. The survey will be programmed to collapse demographic response categories before saving the raw data, such that no cell contains fewer than ten (10) unique individuals, a process tested and successfully implemented on separate DoD SAPRO-funded research. NORC will also conduct a disclosure analysis before data delivery to SAPRO to ensure that the anonymity of the data collection is protected.
Because the surveys are web-based, submission of the anonymous online surveys complete the data collection process. The data input by the respondents is stored in a secure server once the respondents enter their responses. The submitted anonymous survey data will be programmed to save the raw SGID component responses into one confidential data file not accessible to the research team, and the substantive survey responses with the scrambled 10-digit SGID into another confidential data file for analyses by the research team.
Recruitment to the study will be conducted at First-Term Airmen/Guardian Centers (FTAC) where Airmen/Guardians attend to receive trainings. Airmen/Guardians will either receive an email prior to FTAC or use a QR code in a classroom to access the baseline survey. NORC will work closely with DAF base Violence Prevention Integrators (VPI) to provide email language to disseminate the anonymous survey link or a visual projection slide to display in a classroom the QR code to access the baseline survey. All Airmen/Guardians will be recruited to take the baseline survey prior to any FTAC WIT training. The follow up (6-month) survey will be done via email to the study participants by NORC. At baseline we will collect Airmen/Guardians email addresses through direction to an unlinked webpage following submission of a survey response. Email addresses will not be linkable to the SGIDs or to the survey data responses.
The Airmen/Guardians fidelity feedback forms will be administered to a random sample of five Airmen/Guardians receiving the Wingman Intervention Training. This sample will be selected by the DAF staff member implementing the WIT training picking a random point on the roster of WIT attendees and selecting every fifth case until five surveys are completed. Additionally, a single, separate fidelity form will be completed for each WIT session by the WIT implementer.
There are no overt direct benefits to the respondents who compete the survey or fidelity forms. However, benefits may accrue to Airmen/Guardians by encouraging them to be more active bystanders, thereby reducing SH and SA in their environment, and benefits may accrue for those who have been victims of SH and SA because it may provide a safe, anonymous avenue to share their story. Indirect benefits will accrue to DAF HAF/A1Z Resilience Office staff, who can use the results to improve upon their prevention programming, thus supporting more effective DAF prevention programming. Further, Airmen/Guardians may benefit through the improvement of DAF SH and SA prevention programming within the Air Force.
3. Use of Information Technology
One hundred percent (100%) of survey responses will be collected electronically. The baseline and follow-up surveys are web-based, and all respondents will be provided the same anonymous survey links.
4. Non-duplication
The information obtained through this collection is unique and is not already available for use or adaptation from another cleared source.
5. Burden on Small Businesses
This information collection does not impose a significant economic impact on a substantial number of small businesses or entities.
6. Less Frequent Collection
Data collection will occur twice for each respondent, once starting March 2022 and once starting in September 2022 (six months later). Respondents will start the web-based baseline survey (March 2022) with a six-month intake period until September 2022. A 6-month follow-up survey will be released to the sample as of September 2022 (for the earliest set of baseline responders), with data collection for the follow-up closing in March 2023 (allowing for a six-month passage of time between surveys for the later baseline responders).
Two data points are necessary to identify change over time. Surveys are administered prior to the intervention (baseline) and again six months later (follow-up), after experiencing the WIT intervention (for the treatment group; the comparison group will be surveyed on a parallel timeline). Without a follow-up data collection, we will be unable to assess the outcomes associated with the WIT prevention programming. In other words, given the need to identify change over time, two surveys (baseline/follow up) is the least number of surveys possible.
7. Paperwork Reduction Act Guidelines
This collection of information does not require collection to be conducted in a manner inconsistent with the guidelines delineated in 5 CFR 1320.5(d)(2).
8. Consultation and Public Comments
Part A: PUBLIC NOTICE
A 30-Day Federal Register Notice for the collection published on Wednesday, January 26, 2022. The 30-Day FRN citation is 87 FRN 4004.
Part B: CONSULTATION
NORC at the University of Chicago has been contracted by DoD SAPRO to provide program evaluation services. The NORC team has consulted with DoD SAPRO and DAF on WIT program information. NORC held informal discussions with a small sample of DAF personnel to ensure that the recruitment and survey language is understandable and acceptable to the target population of First-Term Airmen/Guardians at DAF. Additionally, the NORC team regularly consults with a small panel of consultants which includes experts in the fields of SH and SA prevention, both within and outside the military context.
9. Gifts or Payment
Respondents will be offered tokens of appreciation for survey participation based upon the guidance of the respective DAF leadership. Specifically:
Participants from DAF will receive an incentive following the baseline and follow-up survey administration. For the baseline survey, participants that complete the baseline survey will receive a $10 gift code to Amazon.com and for the follow-up survey, participants that complete the follow-up survey will receive a $15 gift code to Amazon.com. Collection of the respondents’ contact email for gift code delivery is once again achieved through direction to an unlinked webpage following submission of a survey response. At both timepoints, participants will receive a gift code via the NORC project email ([email protected]) 7-10 days after survey completion.
In SSB, we provide the results from another study of response rates for surveys with service personnel receiving a small monetary incentive compared to service personnel not receiving incentives. The group offered the incentives had a significantly higher response rate.
DAF legal advisors noted no legal objections to the use of tokens of appreciation, and the leadership at DAF have reviewed and approved these token of appreciation structures.
10. Confidentiality
The collection instrument does not require a Privacy Act Statement because the survey instrument is administered anonymously. There is no way to link participants’ survey responses to their identities. Likewise, the Airmen/Guardian program feedback form and the program implementer fidelity feedback form are administered as anonymous survey links (no SGID).
A System of Record Notice (SORN) is not required for this collection because records are not retrievable by personally identifiable information (PII).
A Privacy Impact Assessment (PIA) is not required for this collection because PII is not being collected electronically. Only a single data point, respondent duty email addresses, will be collected. This single data point is not considered PII and is not connected to individual survey responses, which are anonymous.
Anonymous records will be retained during analysis and data will be reported in aggregate to DoD SAPRO. At the time of the study’s close, following the deductive disclosure analyses and appropriate data management, anonymous records will be returned to the Air Force and DoD SAPRO, and all data housed at NORC at the University of Chicago will be destroyed following protocols consistent with federal guidelines.
11. Sensitive Questions
This data collection includes sensitive questions. All survey questions are drawn from existing, validated instruments or have been developed in close consultation with the stakeholders in this program evaluation study, and with input from NORC’s experts and our small panel of experts in this field. These questions have also been assessed by a small group of Air Force personnel to ensure that the recruitment and survey language is understandable and acceptable to the target population of the DAF. The surveys assess topics such as descriptive behavioral norms, experiences of SH and SA, alcohol-related sex expectancies, exposures as a witness to situations that might call for bystander intervention to prevent SH and/or SA, and personal responses when exposed to these situations. The measurement of these outcomes in the baseline and follow-up surveys is necessary to evaluate the impact of the SH and SA prevention programming being conducted at DAF. The survey instrument also includes questions about respondents’ sex, sexual orientation, and race/ethnicity; these sociodemographic responses are programmed to be recoded as binary indicators before the research team sees the data to prevent anyone from determining an individual participant’s identity based on unique characteristics. The two different types of fidelity forms do not contain sensitive items.
12. Respondent Burden and its Labor Costs
Part A: ESTIMATION OF RESPONDENT BURDEN
Collection Instrument(s)
WIT Program Evaluation Survey Baseline
Number of Respondents: 4,000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 1
Response Time: 15-20 minutes
Respondent Burden Hours: 1,167 hours
WIT Program Evaluation Airmen/Guardians Feedback Forms for 15 Treatment Bases
WIT Program Evaluation Implementer Feedback Forms for 15 Treatment Bases
Number of Respondents: 90
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 1
Response Time: 5 minutes
Respondent Burden Hours: 7.5 hours
WIT Program Evaluation Survey Follow-up
Number of Respondents: 4,000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 1
Response Time: 15-20 minutes
Respondent Burden Hours: 1,167 hours
Total Submission Burden (Summation)
Total Number of Survey Respondents: 8,000
Total Number of VPI Feedback Form Respondents: 90
Total Number of Airmen/Guardians Feedback Form Respondents: 450
Total Number of Annual Feedback Form Responses: 1
Total Number of Annual Survey Responses: 2
Total Respondent Burden Hours: 2,379 hours
Part B: LABOR COST OF RESPONDENT BURDEN
Collection Instrument(s)
WIT Program Evaluation Survey Baseline
Number of Total Annual Responses: 1
Response Time: 15-20 minutes
Respondent Hourly Wage: $9.70
Labor Burden per Response: $2.83
WIT Program Evaluation Airmen/Guardians Feedback Forms for 15 Treatment Bases
Number of Total Annual Responses: 1
Response Time: 5 minutes
Respondent Hourly Wage: $9.70
Labor Burden per Response: $0.81
Total Labor Burden: $364.50
WIT Program Evaluation Implementer Feedback Forms for 15 Treatment Bases
Number of Total Annual Responses: 1
Response Time: 5 minutes
Respondent Hourly Wage: $37.09
Labor Burden per Response: $3.09
Total Labor Burden: $278.18
WIT Program Evaluation Survey Follow-up
Number of Total Annual Responses: 1
Response Time: 15-20 minutes
Respondent Hourly Wage: $9.70
Labor Burden per Response: $2.83
Total Labor Burden: $11,319.90
Overall Labor Burden
Total Number of Annual Responses: 2
Total Number of Annual Feedback Form Responses: 1
Total Labor Burden: $23,282.48
Military service pays their service members monthly stipends. The above labor cost of respondent burden is calculated based on the following source (https://www.airforce.com/careers/pay-and-benefits), which indicates that enlisted Airmen/Guardians in the pay grade E1 with less than two years of service are paid at the rate noted above. This calculation assumes a 40-hour work week at this rate of pay. Additionally, Violence Prevention Implementers have varied experience as implementers on the GS-level schedule for labor cost, and the respondent burden for the implementer feedback forms is calculated based on the mid-point of the scale. We used the following source (https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2022/GS_h.pdf), for GS-12 step 5.
13. Respondent Costs Other Than Burden Hour Costs
There are no annualized costs to respondents other than the labor burden costs addressed in Section 12 of this document to complete this collection.
14. Cost to the Federal Government
Part A: LABOR COST TO THE FEDERAL GOVERNMENT
Collection Instrument(s)
WIT Program Evaluation Survey Baseline
Number of Total Annual Responses: 4,000
Processing Time per Response: 0.5 hours (30 minutes)
Hourly Wage of Worker(s) Processing Responses: $138.21
Cost to Process Each Response: $69.73
Total Cost to Process Responses: $278,920.00
WIT Program Evaluation Airmen/Guardians Feedback Forms: 15 Treatment Bases
Number of Total Annual Responses: 450
Processing Time per Response: 0.08 hours (5 minutes)
Hourly Wage of Worker(s) Processing Responses: $138.21
Cost to Process Each Response: $11.06
Total Cost to Process Responses: $4,977.00
WIT Program Evaluation Implementer Feedback Forms for 15 Treatment Bases
Number of Total Annual Responses: 90
Processing Time per Response: 0.08 hours (5 minutes)
Hourly Wage of Worker(s) Processing Responses: $138.21
Cost to Process Each Response: $11.06
Total Cost to Process Responses: $995.40
WIT Program Evaluation Survey Follow-up
Number of Total Annual Responses: 4,000
Processing Time per Response: 0.5 hours (30 minutes)
Hourly Wage of Worker(s) Processing Responses: $140.97
Cost to Process Each Response: $71.13
Total Cost to Process Responses: $284,520.00
Overall Labor Burden to the Federal Government
Total Number of Annual Survey Responses: 8,000
Total Labor Burden: $569,412.40
Part B: OPERATIONAL AND MAINTENANCE COSTS
Cost Categories
Equipment: $0
Printing: $0
Postage: $0
Software Purchases: $0
Licensing Costs: $0
Other: $27,798.47
Total Operational and Maintenance Cost: $27,798.47
Part C: TOTAL COST TO THE FEDERAL GOVERNMENT
Total Labor Cost to the Federal Government: $569,412.40
Total Operational and Maintenance Costs: $27,798.47
Total Cost to the Federal Government: $597,210.87
15. Reasons for Change in Burden
This is a new collection with a new associated burden.
16. Publication of Results
The results will be reported internally for DoD use first and then reviewed for potential publication, pending concurrence between DoD SAPRO and DAF.
17. Non-Display of OMB Expiration Date
We are not seeking approval to omit the display of the expiration date of the OMB approval on the collection instrument.
18. Exceptions to “Certification for Paperwork Reduction Submissions”
We are not requesting any exemptions to the provisions stated in 5 CFR 1320.9.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kaitlin Chiarelli |
| File Modified | 0000-00-00 |
| File Created | 2022-02-02 |