SUPPORTING STATEMENT – PART A
Exploring Civil-Military Views Regarding AI and Related Technologies – OMB Control Number 0704-XXXX
Need for the Information Collection
The U.S. Department of Defense (DoD) is requesting approval from the Office of Management and Budget (OMB) to conduct a survey with members of the software engineering community and employees of defense and aerospace companies. The study will also conduct focus groups with members of the software engineering community and students from computer science programs. This project is funded by the DoD Joint Artificial Intelligence Center (JAIC) and the Office of Net Assessment (ONA), who have contracted with the RAND Corporation, a non-profit research institute, to conduct this study. This data collection will help ensure DoD’s ability to engage with leading private sector technology corporations and their employees.
DoD recognizes that in an information-centric age, information systems and capabilities will be critical to securing the United States and its interests. In order for the DoD to have access to the most advanced technologists in areas such as Artificial Intelligence (AI), it will need to engage with private sector corporations that derive only a small portion of their revenue from contracts with the DoD and do not see themselves as primarily defense contractors. However, recent events have raised concerns about a divide between the views, goals, and motivations from employees at these corporations compared to people employed by or associated with the national security community. For example, Google engineers have forced their company to abandon cooperation with the DoD in applying AI algorithms to analyzing video footage (Project Maven) and to withdraw from bidding on the DoD’s cloud computing contract, the Joint Enterprise Defense Infrastructure (JEDI). This potential divide – if not understood and addressed – may affect DoD’s ability to draw upon the most advanced talent and technology available for its efforts to field game changing improvements in its war fighting ability.
To date, the degree and nature of the differences in outlook between private sector software engineers and their potential customers within the national security communities is not sufficiently understood to provide actionable insights to assist DoD in minimizing the potential risk to defense acquisitions and capabilities. While many commentators and reporters have published on this topic, their analysis has been primarily based on either the preconceptions of the authors or anecdotes about the opinions of individual engineers. It is unclear to what extent these pieces capture the views of software engineers working for these companies or working in the industry overall.
Consequently, the only way to gain a comprehensive understanding of the views of this community, how their views are evolving, and how they compare to the broader population is by conducting a large-scale survey of software engineers capable of performing this type of work. Responses from these engineers can then be compared to those from others in the software engineering community.
This collection is authorized by DoD Instruction 5111.11.
Use of the Information
In order to learn the opinions of software engineers about issues surrounding national security, the U.S. Federal Government, and the DoD, RAND will administer a survey to the population of interest (private sector software engineers), as well as to comparison groups drawn from the defense and aerospace industry. In addition to the survey, RAND will conduct focus groups with members of the software engineering community and current students of top Computer Science (CS) university programs, as these students may have a similar career trajectory to software engineers and tech company employees. Focus groups will be used to gather qualitative, in depth responses that cannot be collected in a survey.
The populations of greatest interest to DoD are software engineers at leading American software companies, such as the so-called FAANG corporations (Facebook, Apple, Amazon, Netflix, and Google) and other companies of similar size and capability. Employees of these corporations have the greatest experience with state-of-the-art software technologies such as cloud computing and these corporations have been responsible for most of the recent advances in AI. However, employees of these companies also have little, if any, contact with the national security community and some have strong ethical objections to building the kinds of software applications that are vital to the future of U.S. national security. Understanding how these individuals view ethical questions involving the use of AI in national security applications is a priority to our sponsor.
In addition to members of the software engineering community at leading technology companies, RAND will also administer the survey to members of the software engineering community more broadly (e.g., alumni of leading computer science programs and employees with a job title of “software engineer”) and engineers working in the defense and aerospace industry. Comparing these various groups will help establish if primary survey populations have unique objections stemming from their familiarity with the technologies necessary to create AI applications. Conversely, it will also explore whether their ethical views are similar to members of the software engineering community who work elsewhere or to engineers who have chosen to work closely with the Defense Department.
Survey respondents will complete the survey by web. Sampled participants will receive an email invitation that explains the purpose of the study and include a link to the survey instrument. The survey data collection will include follow-up with non-respondents to maximize response rates. For instance, the contractor will send prompting emails to selected sampled individuals who have not responded to requests. After the survey is completed, data will be automatically transmitted to the contractor for analysis.
In addition to the survey, the contractor will conduct focus groups with members of the software engineering community and current students at top ranked CS departments. Focus groups will be used to supplement the survey responses and dive deeper into the specific reasoning and perspectives behind trends that emerge from analysis of the data. While the survey will allow for quantitative analysis, the focus groups and interviews will allow us to obtain a deeper understanding of the factors that influence an individual’s decision about working on specific AI projects or for specific organizations. For example, the survey instrument measures respondent trust in both government and non-government organizations. While this is important information, the survey cannot capture the reasoning behind these opinions, or the personal experiences and stories that led to these opinions. In addition, anecdotes and explanations captured through qualitative means can be especially useful if survey response data is ambiguous or not as expected.
Focus group candidates will be invited to participate by email or advertisement. For focus groups convened at industry conferences, we will provide text for the focus group invitation, which will include the date, time, location, and other pertinent details, to conference organizers for distribution with other conference materials. For focus groups convened with students, we will similarly send information about the focus group to related listservs and social media sites for posting, and may also print flyers with focus group details for posting or distribution. All materials will include the RAND team’s contact information and will invite interested students to contact RAND. Additionally, we may use commercial focus group vendors to identify and recruit participants. These vendors have databases of individuals and will call people and screen them for eligibility and interest in the focus group.
Follow-up with individuals who have not responded to the initial invitation will be by email. The focus groups will last 90 minutes; notes will be taken and transcripts of audio recordings will be created for analysis. Once data collection is complete, respondents will have no further contact from the project team.
Use of Information Technology
All survey responses will be collected electronically. The survey will utilize a web-based instrument for data collection. The web-survey will be formatted to be easy to read and navigate. In addition, a Records Management System (RMS) will be used to manage the survey sample, document interim and final case status codes, and generate progress reports. The RMS will track case status in all modes and will document the actions and outcomes for each case in the sample for all phases of activity.
Non-duplication
The information obtained through this collection is unique and is not already available for use or adaptation from another cleared source.
Burden on Small Businesses
No respondents are expected to be small businesses or other small entities. This information collection does not impose a significant economic impact on a substantial number of small businesses or entities.
Less Frequent Collection
The survey and focus groups will be a one-time data collection effort. In the absence of this data collection, we will not be able to assess the extent to which DoD will encounter difficulty in acquiring leading edge technologies and capabilities for DoD missions.
Paperwork Reduction Act Guidelines
This collection of information does not require collection to be conducted in a manner inconsistent with the guidelines delineated in 5 CFR 1320.5(d)(2).
Consultation and Public Comments
Part A: PUBLIC NOTICE
A 60-Day Federal Register Notice for the collection was published on Thursday, January 16, 2020. The 60-Day FRN citation is 85 FR 2726.
No comments were received during the 60-Day Comment Period.
A 30-Day Federal Register Notice for the collection was published on Monday, July 27, 2020. The 30-Day FRN citation is 85 FRN 45203.
Part B: CONSULTATION
Staff from the RAND Corporation were consulted on this project:
James Ryseff, M.S.
Eric William Landree, Ph.D.
Madhumita (Bonnie) Ghosh Dastidar, Ph.D.
Gifts or Payment
No payments or gifts are being offered to survey respondents or focus group participants..
Confidentiality
This collection instrument requires a Privacy Act Statement since it collects PII. It is displayed to the user immediately before they begin answering survey questions on the first page of the survey.
A System of Record Notice is not required for this collection because the records are not retrievable by PII.
A draft copy of the PIA, Exploring Views about the Use of Artificial Intelligence Survey, has been provided with this package for OMB’s review.
All information collected by this survey will be destroyed once the final report has been written. This will occur no later than twelve months after the Office of Management and Budget approves the survey and the collection period begins.
Sensitive Questions
No questions considered sensitive are being asked in this collection.
Respondent Burden and its Labor Costs
Estimation of Respondent Burden
Survey of Members of Software Engineering Community
Number of Respondents: 3000
Number of Responses Per Respondent: 1
Number of Total Annual Responses:3,000
Response Time: .25 hours
Respondent Burden Hours: 750 hours
Survey of Members of the Defense and Aerospace Industry
Number of Respondents: 1000
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 1,000
Response Time: .25 hours
Respondent Burden Hours: 250 hours
Focus Groups, Screening Calls with Students
Number of Respondents: 80
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 80
Response Time: .25 hours
Respondent Burden Hours: 20 hours
Focus Groups, Screening Calls with Members of Software Engineering Community
Number of Respondents: 60
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 60
Response Time: .25 hours
Respondent Burden Hours: 15 hours
Focus Groups with Students
Number of Respondents: 40
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 40
Response Time: 1.5 hours
Respondent Burden Hours: 60 hours
Focus Groups with Members of Software Engineering Community
Number of Respondents: 30
Number of Responses Per Respondent: 1
Number of Total Annual Responses: 30
Response Time: 1.5 hours
Respondent Burden Hours: 45 hours
Total Submission Burden
Total Number of Respondents: 4,210
Total Number of Annual Responses: 4,210
Total Respondent Burden Hours: 1,140 hours
Labor Cost of Respondent Burden
Survey of Members of Software Engineering Community
Number of Total Annual Responses: 3000
Response Time: .25 hours
Respondent Hourly Wage: $50.23
Labor Burden per Response: $12.56
Total Labor Burden: $37,672.50
Survey of Members of Defense and Aerospace Industry
Number of Total Annual Responses: 1000
Response Time: .25 hours
Respondent Hourly Wage: $50.23
Labor Burden per Response: $12.56
Total Labor Burden: $12,557.50
Focus Groups, Screening Calls with Students
Number of Total Annual Responses: 80
Response Time: .25 hours
Respondent Hourly Wage: $24.98
Labor Burden per Response: $6.25
Total Labor Burden: $499.60
Focus Groups, Screening Calls with Members of Software Engineering Community
Number of Total Annual Responses: 60
Response Time: .25 hours
Respondent Hourly Wage: $50.23
Labor Burden per Response: $12.56
Total Labor Burden: $753.60
Focus Groups with Students
Number of Total Annual Responses: 40
Response Time: 1.5 hours
Respondent Hourly Wage: $24.98
Labor Burden per Response: $37.47
Total Labor Burden: $1498.80
Focus Groups with Members of Software Engineering Community
Number of Total Annual Responses: 30
Response Time: 1.5 hours
Respondent Hourly Wage: $50.23
Labor Burden per Response: $75.35
Total Labor Burden: $2,260.50
Overall Labor Burden
Total Number of Annual Responses: 4,210
Total Labor Burden: $55,242.50
Average hourly wage was derived from the Bureau of Labor and Statistics (BLS) May 2018 Occupational Employment and Wage Estimates. The rate used for members of the software engineering community is the mean hourly wage of Software Developers and Programmers (15-1130, $50.23). For students, we used the mean hourly rate for all occupations (00–0000, $24.98). (http://www.bls.gov/oes/current/oes_nat.htm).
Respondent Costs Other Than Burden Hour Costs
There are no annualized costs to respondents other than the labor burden costs addressed in Section 12 of this document to complete this collection.
Cost to the Federal Government
JAIC is supporting this data collection and analysis of data as part of the contract with the RAND Corporation. The estimated cost for this work, including design, fieldwork, and analysis, will be $650,000 over 18 months ($433,333 per year).
Reasons for Change in Burden
This is a new collection with a new associated burden.
Publication of Results
RAND will publish a public report at the conclusion of the survey. This topic is of great interest to a variety of audiences and publishing results will advance the state of the public debate over these topics, as well as helping to promote understanding of ethical issues regarding AI generally and the specific ethical issues involving issues in national security specifically.
The expected title of the publication will be “Exploring Views about the Use of Artificial Intelligence” and will be completed approximately seven months after we receive OMB approval for the survey. We expect to publish the completed report within 12 months after receiving OMB approval for the survey.
RAND does not anticipate doing quantitative analysis with the information gleaned from the focus groups. Rather, this information will be summarized in paragraph form and used to supplement the quantitative information derived from the surveys. The following table overviews the publication and project schedule.
Tasks |
|
Month 1 |
Month 2 |
Month 3 |
Month 4 |
Month 5 |
Month 6 |
Month 7 |
OMB Approval |
X |
|
|
|
|
|
|
|
Prepare Online Survey |
|
X |
X |
|
|
|
|
|
Conduct Survey and Focus Groups |
|
|
X |
X |
X |
X |
|
|
Data Analysis |
|
|
|
X |
X |
X |
X |
|
Final Report |
|
|
|
|
|
|
X |
X |
Non-Display of OMB Expiration Date
We are not seeking approval to omit the display of the expiration date of the OMB approval on the collection instrument.
Exceptions to “Certification for Paperwork Reduction Submissions”
We are not requesting any exemptions to the provisions stated in 5 CFR 1320.9.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kaitlin Chiarelli |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |