HOUSE_OVERSIGHT_014706.jpg

Esa Origins 7 February 24 ~ 26, 2017 PROJECT An Origins Project Scientific Workshop Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes ARIZONA STATE UNIVERSITY 4) Al, CYBERSECURITY, AND Al ATTACK SURFACES Al Attacks on Computing Systems, Devices, Infrastructure (Contributions by Kathleen Fisher, John Launchbury, Ashish Kapoor, Sean, Shahar, Jeff Coleman and others) Al will be used in new ways to enhance cyberwarfare. Targets could be either purely computational, aimed at the bringing down of computing systems, the stealing of stored information, of gaining access to monitoring activity and information streams. However, we are more likely to see potentially even more costly attacks involving a combination of cyber and physical systems, e.g., uranium enrichment plants, automated flight systems, weapon systems, automated driving systems, healthcare equipment, oil refineries, or the large swaths of the power grid of the US or other countries. Cyberwarfare is a domain in which the use of Al is inevitable. Attacks and/or responses are likely to happen at computing rather than human speeds. As soon as one side has autonomous cyber warriors systems (ACWs), other actors will have to adapt similar offensive or new defensive technologies. Given this context, imagine building an ACW designed to seek, disrupt, and destroy within high-value adversary networks and systems. The ACW has to be able to observe network behavior to build situational awareness, find places to hide, create exploits to pivot to new places, build a map and use it to navigate complex networks, find high-value information, and identify targets to disable or from which to extract information. Because high-value adversary networks are likely to be relatively isolated, the ACW will have very limited opportunities for external command and control communication, so it will need to make many decisions in isolation. It will read information it finds, build a model of adversarial intent, and then invent ways to disrupt that intent. Establishing the initial access to the high-value network is likely challenging, so the ACW will spawn and spread to ensure that it can reconstruct itself if an active part is observed and destroyed. The ACW may also create disguised caches of specific capabilities so that it can construct new mission-oriented functionality from pieces. It will morph its active form so that defenses will have a hard time finding it. It will inject itself into trusted binaries so that its behavior is difficult to distinguish from legitimate applications. The mission of the ACW will likely be defined in flexible terms because the human handlers will have only limited information when it is deployed. The ACW will be designed to seek opportunities to communicate with its human handlers, but it will also be designed to act autonomously if it observes triggering behavior in the adversary’s systems. It may try to distinguish training states from active warfare states on adversary systems. The creators of the ACW will have had to trade off the likely effectiveness of the ACW versus the cost of premature action. Awareness of the adversary’s systems will necessarily be limited in accuracy because it only gets a worm’s eye view of the network from the portions of the system it has been able to compromise. Once the ACW triggers an active mission, it will 10 HOUSE_OVERSIGHT_014706