HOUSE_OVERSIGHT_011294.jpg

Esa Origins 7 February 24 - 26, 2017 PROJECT An Origins Project Scientific Workshop Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes ARIZONA STATE UNIVERSITY portions of the system it has been able to compromise. Once the ACW triggers an active mission, it will work to degrade or destroy specific functionality (e.g., rewriting network routine tables, replacing plans, changing target information). Once the technology for ACWs exists for military targets, it seems likely there will be cross over into civilian use. Such technology could be deployed against law enforcement targets to disrupt criminal investigations, against banks to steal financial assets, or against companies to steal intellectual property. As they spread into these more general targets, the effects of ACWs might become less predictable. If an ACW incorrectly assesses the situation, it might end up taking down a flight control center or a stock exchange, for example. SOURCES The initial development of ACWs will likely be done by nation states with good intentions, i.e., securing the national interests. (Although what is in one country’s national interest may well not be in the national interest of other countries). The shared existence of such technology might serve as a deterrent against their use by anyone in much the same way that nuclear weapons have served as a deterrent, although ACWs would likely have to be used to devastating effect first to establish their efficacy and threat. However, once the technology exists, it would be very difficult to keep it out of the hands of people with malicious intent (criminals, terrorists, and rogue nation states). It is also the case that the technology has the potential to cause significant collateral damage even if its use was originally well intentioned because it can be difficult to distinguish civilian from military targets in cyberspace. PERSISTENCE Characteristics engineered into the ACW are likely to make it persistent and hard to find as it is designed to infiltrate adversary systems and hide from detection. Once released and active in the open Internet, it may be economically impossible to destroy and remove. OBSERVABILITY Both implicit/insidious and explicit/obvious costly outcomes are conceivable. An ACW could make subtle changes to systems that cause adverse outcomes while hiding its tracks, making it extremely difficult to determine why something has gone wrong or even that something has gone wrong. Attacks that impact the physical world would be harder to mask, but it might still be possible to hide the role of the ACW in the attack. TIME FRAME It seems likely we would start to see ACWs in less than 15 years. Initial steps along these lines are already taking place; see DARPA’s Cyber Grand Challenge, which took place in August 2016 in Las Vegas. The Cyber Reasoning Systems (CRS) that competed in that event are still primitive, the first of their kind. The team that won the competition came in last in the human-league capture-the-flag tournament that happened immediately after. The situation is likely analogous to what we have seen 11 HOUSE_OVERSIGHT_011294