Autonomous Weapons

Risk Assessment

Overview

Lethal autonomous weapons systems (LAWS) represent one of the most immediate and consequential applications of artificial intelligence in military contexts. These systems can select, prioritize, and engage human targets without direct human authorization for each lethal action. Unlike science fiction depictions, autonomous weapons are not futuristic possibilities—they are present battlefield realities that have already claimed human lives and fundamentally altered the character of modern warfare.

The significance of autonomous weapons extends far beyond military considerations. They represent a profound shift in how decisions about human life and death are made, potentially transferring moral agency from humans to algorithms. This transformation raises fundamental questions about accountability, proportionality, and the nature of warfare itself. The speed of autonomous systems—operating in milliseconds rather than the seconds or minutes required for human decision-making—creates new dynamics where conflicts could escalate beyond human comprehension or control.

Current evidence indicates that autonomous weapons lower barriers to armed conflict by reducing the human and financial costs of military operations. They enable continuous, sustained operations without human fatigue, potentially making warfare more frequent and prolonged. Most concerningly, as these capabilities proliferate to non-state actors and less stable regions, they threaten to democratize lethal force in ways that could destabilize international security.

Global Market and Investment

The autonomous weapons sector has grown into a major defense industry segment, with substantial government and private investment accelerating development across all major military powers.

The market is driven by increasing defense budgets, escalating geopolitical tensions, and the demonstrated effectiveness of autonomous systems in Ukraine. North America accounts for approximately 28% of the global market, while Asia-Pacific represents the largest regional market. The U.S. Department of Defense has allocated over 1.2 billion USD in its 2024 budget specifically for development, testing, and deployment of AI-powered autonomous weapon systems.

The Autonomy Spectrum and Human Control

Modern weapons systems exist along a complex spectrum of human control, making simple binary classifications inadequate for policy or ethical analysis. At the most restrictive end, human-operated systems require direct human control for target identification, selection, and engagement—essentially sophisticated tools that amplify human capabilities without substituting human judgment.

Semi-autonomous systems represent the current mainstream of military AI, where humans delegate certain functions to algorithms while retaining ultimate authority over lethal decisions. These "human-in-the-loop" systems present targeting recommendations and require explicit human authorization before firing. However, the practical meaning of "human control" becomes murky when systems present complex information that humans cannot fully process, or when operational tempo demands decisions faster than human cognitive speeds allow.

Human-supervised autonomous weapons, sometimes called "human-on-the-loop" systems, operate autonomously unless a human operator actively intervenes to abort an engagement. These systems fundamentally reverse the authorization paradigm—instead of requiring human approval to act, they require human action to stop. This seemingly subtle distinction has profound implications for moral responsibility and operational dynamics, particularly when multiple autonomous systems operate simultaneously at speeds that overwhelm human supervisory capacity.

Fully autonomous weapons systems can identify, prioritize, track, and engage targets based entirely on their programming and sensor inputs, without any human involvement in individual targeting decisions. While no military openly admits to deploying such systems against human targets, the technical capabilities exist, and the operational pressures of modern warfare increasingly push military systems toward this level of autonomy.

flowchart TD
  subgraph Control["Human Control Spectrum"]
      A[Human-Operated] --> B[Semi-Autonomous]
      B --> C[Human-Supervised]
      C --> D[Fully Autonomous]
  end

  subgraph Decision["Decision Authority"]
      A1["Human selects<br/>Human engages"]
      B1["AI recommends<br/>Human approves"]
      C1["AI acts<br/>Human can abort"]
      D1["AI acts<br/>No human required"]
  end

  A -.-> A1
  B -.-> B1
  C -.-> C1
  D -.-> D1

  subgraph Examples["Current Examples"]
      E1["Guided missiles"]
      E2["Iron Dome"]
      E3["Loitering munitions"]
      E4["Kargu-2 (alleged)"]
  end

  A1 -.-> E1
  B1 -.-> E2
  C1 -.-> E3
  D1 -.-> E4

  style D fill:#ff6b6b
  style D1 fill:#ff6b6b
  style E4 fill:#ff6b6b

The spectrum above illustrates the progression from human-controlled to fully autonomous systems. The transition from "human-in-the-loop" to "human-on-the-loop" represents a fundamental shift in authorization paradigms, with significant implications for accountability and escalation dynamics.

Evidence of Battlefield Deployment

The transition from theoretical possibility to battlefield reality has occurred with remarkable speed. The March 2020 incident in Libya, documented in a UN Security Council Panel of Experts report↗🔗 webUN Security Council Panel of Experts reportThis is a key real-world precedent for LAWS deployment, directly relevant to debates on autonomous weapons governance, human-in-the-loop requirements, and the urgency of international AI arms control treaties.This Popular Mechanics article analyzes a UN Security Council Panel of Experts report documenting the first known instance of autonomous weapons systems independently targeting ...governancemilitary-aiarms-controlpolicy+4Source ↗ (S/2021/229), marked a watershed moment when a Turkish-supplied Kargu-2 loitering munition↗🔗 webKargu-2 loitering munitionRelevant to AI safety discussions around autonomous weapons governance and accountability; provides a concrete case study of deployed lethal autonomous systems and the gaps in existing legal frameworks for attributing responsibility.This analysis examines the legal and ethical dimensions of the Kargu-2 loitering munition following a 2021 UN report suggesting it may have been used autonomously in Libya. The ...governancemilitary-aiarms-controlpolicy+3Source ↗ allegedly engaged human targets autonomously, without remote pilot control or explicit targeting commands. According to the UN report, the drones "were programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true 'fire, forget and find' capability."

Ukraine's conflict has become what CSIS analysts↗🔗 web★★★★☆CSISUkraine's Future Vision and Current Capabilities for Waging AI-Enabled Autonomous WarfareThis CSIS report is relevant to AI safety discussions around lethal autonomous weapons systems (LAWS), as Ukraine's conflict provides a real-world precedent for AI-enabled autonomous warfare that may shape future governance and arms control debates.A CSIS report by Kateryna Bondar analyzing Ukraine's strategic vision and current capabilities for AI-enabled autonomous warfare, covering AI applications in ISR, automatic targ...military-aigovernancepolicydeployment+4Source ↗ describe as "the Silicon Valley of offensive AI." The December 2024 first fully unmanned operation near Lyptsi↗🔗 web★★★★☆Atlantic CouncilDecember 2024 first fully unmanned operation near LyptsiRelevant to AI safety discussions around autonomous weapons and lethal autonomous systems (LAWS); illustrates real-world deployment of AI-driven military drones with increasing autonomy in an active conflict, raising questions about oversight and control.This Atlantic Council article outlines Ukraine's defense technology priorities for 2025, focusing on AI-integrated drones, interceptor systems, long-range strike capabilities, a...military-aiautonomous-weaponsdeploymentcapabilities+3Source ↗, north of Kharkiv, represented a qualitative escalation - an entire military operation conducted exclusively by autonomous ground and aerial systems without human pilots in direct control.

Ukraine Drone Production and AI Integration (2024)

AI Effectiveness Data

The performance differential between manual and AI-guided drones demonstrates the military advantage driving autonomous weapons adoption:

This 4-8x improvement in efficiency creates powerful incentives for autonomous systems adoption, particularly as electronic warfare degrades manual drone control links.

Commercial proliferation has made autonomous weapons capabilities accessible to non-state actors and smaller militaries. The underlying technologies - computer vision, GPS navigation, and basic AI algorithms - are increasingly available through civilian supply chains. NORDA Dynamics, a Ukrainian company, has sold over 15,000 units of its automated targeting software, with over 10,000 already delivered to drone manufacturers. This technological democratization means that autonomous weapons capabilities are spreading far beyond the advanced militaries that initially developed them.

Safety and Reliability Concerns

Autonomous weapons systems operate in environments specifically designed to defeat them through deception, jamming, and spoofing. Unlike civilian AI applications where failures typically result in inconvenience or financial loss, autonomous weapons failures can cause mass casualties or escalate conflicts. The adversarial nature of warfare means that opponents actively work to exploit vulnerabilities in autonomous systems, creating failure modes that may be impossible to anticipate during development and testing.

Technical reliability remains a fundamental concern. Military AI systems must operate across diverse environments, against adaptive adversaries, with limited opportunities for software updates or repairs. Computer vision systems can be confused by camouflage, weather conditions, or deliberate deception. GPS systems can be jammed or spoofed. Communication links can be severed. Each of these vulnerabilities becomes potentially lethal when embedded in autonomous weapons.

The verification and validation challenges for autonomous weapons exceed those of any previous military technology. Unlike conventional weapons with predictable ballistics and blast effects, AI systems exhibit emergent behaviors that cannot be fully tested in advance. The space of possible scenarios is effectively infinite, making comprehensive testing impossible. This uncertainty becomes particularly problematic when systems encounter edge cases or adversarial conditions not represented in their training data.

Attribution and accountability present additional challenges. When an autonomous system causes unintended casualties or commits what would constitute a war crime if performed by humans, determining responsibility becomes complex. Is the blame with the programmer, the commanding officer who deployed the system, the manufacturer, or the political leadership that authorized its use? This accountability gap could create practical immunity for war crimes conducted through algorithmic intermediaries.

Escalation Dynamics and Strategic Stability

Autonomous weapons fundamentally alter the tempo and character of military conflict. Human decision-making operates on timescales of seconds to minutes, while autonomous systems can complete observe-orient-decide-act cycles in milliseconds. This speed differential creates new categories of conflict where human commanders may find themselves managing wars that unfold too quickly for meaningful human control or intervention.

Flash wars represent a new category of potential conflict where autonomous systems from different militaries interact at machine speeds, potentially escalating from peaceful coexistence to full conflict before human operators can intervene. These scenarios become particularly dangerous when combined with nuclear weapons systems, where autonomous early warning systems might recommend preemptive strikes based on algorithmic analysis of threatening patterns.

The proliferation of autonomous weapons lowers traditional barriers to armed conflict. Historically, the human cost of military operations provided a natural brake on aggressive policies—populations and leaders had to weigh potential gains against the lives of their own soldiers. Autonomous systems reduce these human costs for the attacking side, potentially making military action more politically palatable and increasing the frequency of armed conflicts.

Deterrence relationships become unstable when opponents cannot clearly understand each other's autonomous capabilities or decision algorithms. Traditional deterrence relies on predictable rational responses, but autonomous systems may exhibit behaviors that human opponents cannot anticipate or interpret correctly. This uncertainty could lead to overreaction during crises or failure to recognize escalatory signals.

International Governance Efforts

International efforts to govern autonomous weapons have struggled to keep pace with technological development and military deployment. The UN Convention on Certain Conventional Weapons (CCW) has hosted discussions on lethal autonomous weapons systems since May 2014, but has failed to produce binding agreements due to its consensus-based decision-making process. As Human Rights Watch notes↗🔗 web★★★★☆Human Rights WatchHuman Rights Watch notesRelevant to AI governance discussions around autonomous weapons; represents civil society advocacy perspective following a key 2024 UN vote, useful for tracking international policy developments on lethal autonomous systems.Human Rights Watch calls on states to pursue a binding international treaty on autonomous weapons systems following a UN General Assembly vote, arguing that existing internation...governancepolicymilitary-aiarms-control+3Source ↗, "a handful of major military powers - notably India, Israel, Russia, and the United States - have exploited this process to repeatedly block proposals to negotiate a legally binding instrument."

UN General Assembly Voting on LAWS (2023-2024)

The December 2024 UN General Assembly resolution↗🔗 webDecember 2024 UN General Assembly resolutionPublished by the American Society of International Law in January 2025, this piece provides legal analysis of the landmark UNGA LAWS resolution and the international regulatory landscape, relevant to AI governance and autonomous weapons policy discussions.This ASIL Insight analyzes the December 2024 UN General Assembly resolution on lethal autonomous weapons systems (LAWS), which passed 166-3, and examines momentum toward a new i...governancepolicymilitary-aiarms-control+5Source ↗ represents the strongest international statement to date, acknowledging the "negative consequences and impact of autonomous weapon systems on global security and regional and international stability, including the risk of an emerging arms race." However, it lacks enforcement mechanisms and does not mandate treaty negotiations due to U.S. opposition. The resolution approves "open informal consultations" in New York during 2025.

National Positions on LAWS Governance

Three core positions have emerged in international negotiations, as analyzed by the Lieber Institute↗🔗 webFuture Warfare: National Positions and Governance of Lethal Autonomous Weapons SystemsPublished by the Lieber Institute at West Point, this piece is relevant to AI safety researchers interested in international governance of autonomous weapons, a key near-term policy challenge intersecting AI capabilities and international security law.A Lieber Institute analysis examining how different nations are positioning themselves on lethal autonomous weapons systems (LAWS) and the international governance frameworks be...governancepolicymilitary-aiarms-control+4Source ↗:

The Campaign to Stop Killer Robots↗🔗 webStop Killer Robots Campaign VideosThis is the homepage of the leading civil society coalition opposing lethal autonomous weapons; relevant to AI governance, military AI risk, and international policy debates around autonomous systems.Stop Killer Robots is a global coalition campaign advocating for a ban on fully autonomous weapons systems (lethal autonomous weapons or 'killer robots'). The campaign pushes fo...governancepolicyai-safetyexistential-risk+3Source ↗, launched in April 2013, has mobilized civil society organizations, Nobel laureates, and tech industry leaders to advocate for preemptive bans. In September 2024, the UN Secretary-General and ICRC issued a joint appeal↗🔗 web★★★★☆Human Rights WatchUN Secretary-General and ICRC issued a joint appealCovers a key 2024 UN policy development on lethal autonomous weapons systems, relevant to AI governance discussions about maintaining human oversight and control over AI-enabled military systems.Human Rights Watch reports on a UN Secretary-General report released August 2024 calling for an international treaty by 2026 to prohibit lethal autonomous weapons systems (LAWS)...governancepolicymilitary-aiarms-control+4Source ↗ calling for states to negotiate new law by 2026, warning that "time is running out for the international community to take preventive action."

Current Military Programs and Capabilities

Major military powers have invested heavily in autonomous weapons capabilities while maintaining official policies requiring human control over lethal decisions. The U.S. Department of Defense Directive 3000.09↗🔗 webU.S. Department of Defense Directive 3000.09This is the primary U.S. government regulatory document governing autonomous weapons; it is a key reference for understanding how the DoD formally approaches human control, oversight, and accountability in AI-enabled weapon systems.The U.S. Department of Defense Directive 3000.09 establishes binding policy and responsibilities for the development, testing, and employment of autonomous and semi-autonomous w...governancepolicymilitary-aideployment+4Source ↗, updated in January 2023, defines LAWS as "weapon system[s] that, once activated, can select and engage targets without further intervention by a human operator." The directive requires that systems be designed to "allow commanders and operators to exercise appropriate levels of human judgment over the use of force," though Human Rights Watch↗🔗 web★★★★☆Human Rights WatchReview of the 2023 US Policy on Autonomy in Weapons SystemsRelevant to AI governance discussions on autonomous weapons regulation; provides a civil society critique of US federal policy and its divergence from emerging international norms on lethal autonomous systems.Human Rights Watch analyzes the 2023 DoD Directive 3000.09 on autonomous weapons systems, finding it an inadequate revision of the 2012 predecessor that fails to close key looph...governancepolicymilitary-aiarms-control+3Source ↗ notes it "misses an opportunity to address its shortcomings" and allows certain waivers to senior reviews.

Major Power LAWS Programs

Per Section 1066 of the FY2025 NDAA↗🏛️ government★★★★★US CongressSection 1066 of the FY2025 NDAAAn authoritative U.S. government reference document for policymakers and researchers tracking the legal and strategic status of autonomous weapons; particularly relevant for AI governance discussions around lethal autonomous systems and human oversight requirements.This Congressional Research Service primer explains U.S. policy on lethal autonomous weapon systems (LAWS), clarifying that U.S. policy does not prohibit their development or em...governancepolicymilitary-aiarms-control+4Source ↗, the U.S. Secretary of Defense must now submit annual reports on "the approval and deployment of lethal autonomous weapon systems" to congressional defense committees through December 31, 2029.

Russian military doctrine explicitly embraces autonomous weapons development, with extensive Ukraine deployment of Lancet loitering munitions and AI-enhanced targeting systems that identify and prioritize targets with minimal human oversight. Chinese military development focuses heavily on "intelligent" weapons systems, with the PLA's strategic vision emphasizing AI advantages to overcome numerical disadvantages. Israeli defense companies have pioneered semi-autonomous technologies, including Iron Dome which operates autonomously to intercept incoming projectiles.

Trajectory and Future Developments

Timeline Projections

The next 1-2 years will see continued proliferation of semi-autonomous systems with increasing levels of independence. In December 2024, Helsing announced↗🔗 web★★★★☆Atlantic CouncilDecember 2024 first fully unmanned operation near LyptsiRelevant to AI safety discussions around autonomous weapons and lethal autonomous systems (LAWS); illustrates real-world deployment of AI-driven military drones with increasing autonomy in an active conflict, raising questions about oversight and control.This Atlantic Council article outlines Ukraine's defense technology priorities for 2025, focusing on AI-integrated drones, interceptor systems, long-range strike capabilities, a...military-aiautonomous-weaponsdeploymentcapabilities+3Source ↗ that the first few hundred of almost 4,000 AI-equipped HX-2 Karma unmanned aerial vehicles were set for delivery to Ukraine, representing a significant scaling of AI-enabled systems.

Medium-term developments over 2-5 years will likely include swarm capabilities where multiple autonomous systems coordinate actions without human oversight. These swarms could overwhelm traditional defenses and make meaningful human control practically impossible when hundreds or thousands of autonomous systems operate simultaneously. Integration with broader military AI systems will create autonomous kill chains where human oversight becomes limited to high-level policy decisions.

flowchart LR
  subgraph Near["2025-2026"]
      A1[Semi-autonomous<br/>proliferation]
      A2[First swarm<br/>operations]
  end

  subgraph Mid["2027-2030"]
      B1[Autonomous<br/>kill chains]
      B2[Non-state<br/>acquisition]
  end

  subgraph Long["2030+"]
      C1[Fully autonomous<br/>as norm]
      C2[Governance<br/>regime?]
  end

  A1 --> B1
  A2 --> B1
  B1 --> C1
  B2 --> C1
  C1 -.-> C2

  style C1 fill:#ff6b6b
  style B2 fill:#ffa500

Regulatory capture represents a significant risk as defense contractors with autonomous weapons investments gain influence over policy decisions. The CCW Group of Governmental Experts meets in March and September 2025↗🔗 webMarch and September 2025Official UN UNODA page for the 2025 CCW GGE LAWS sessions; essential reference for tracking international diplomatic progress (or stagnation) on autonomous weapons regulation, directly relevant to AI governance and safety researchers monitoring military AI policy.This page covers the 2025 meeting sessions of the UN Convention on Certain Conventional Weapons (CCW) Group of Governmental Experts (GGE) on Lethal Autonomous Weapons Systems (L...governancepolicycoordinationmilitary-ai+4Source ↗, with a 2026 CCW review conference as the deadline for recommendations. However, the consensus requirement and major power opposition make binding agreements unlikely in this timeframe.

Critical Uncertainties and Research Gaps

Key Research Questions

The fundamental question of whether meaningful human control is technically possible in modern warfare remains unresolved. As autonomous systems operate at increasingly fast speeds and in environments where human communication is degraded, the practical meaning of human oversight becomes questionable. The accountability gap presents particular challenges - as Austria, Costa Rica, and other states have noted at the CCW, "it is unclear who could be held legally responsible if such systems violate international humanitarian law or human rights law."

The behavioral characteristics of autonomous weapons in complex, adversarial environments remain poorly understood. Current testing occurs primarily in controlled scenarios that may not represent the chaos, uncertainty, and deliberate deception of actual warfare. The space of possible scenarios is effectively infinite, making comprehensive testing impossible.

International law adaptation presents unresolved challenges. Traditional concepts like distinction between combatants and civilians, proportionality in attacks, and precautions in attack assume human decision-makers capable of contextual judgment. The August 2024 UN Secretary-General report addressed these challenges "from humanitarian, legal, security, technological and ethical perspectives," reflecting 58 submissions from over 73 countries.

Long-term strategic stability with widespread autonomous weapons deployment remains theoretically uncertain. Game theory and strategic studies have not fully explored how deterrence, escalation dynamics, and crisis stability change when military systems can interact autonomously at machine speeds. The emergence of "flash war" scenarios - where autonomous systems escalate faster than human intervention is possible - represents a novel category of strategic risk requiring urgent research attention.