Another Kind of AI x-Risk: The Accumulative AI x-Risk Hypothesis
paperAuthor
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
A conceptually important paper for broadening x-risk discourse beyond superintelligence scenarios; particularly relevant to discussions of societal-scale systemic risk, governance strategy, and bridging near-term and long-term AI safety communities.
Paper Details
Metadata
Abstract
The conventional discourse on existential risks (x-risks) from AI typically focuses on abrupt, dire events caused by advanced AI systems, particularly those that might achieve or surpass human-level intelligence. These events have severe consequences that either lead to human extinction or irreversibly cripple human civilization to a point beyond recovery. This discourse, however, often neglects the serious possibility of AI x-risks manifesting incrementally through a series of smaller yet interconnected disruptions, gradually crossing critical thresholds over time. This paper contrasts the conventional "decisive AI x-risk hypothesis" with an "accumulative AI x-risk hypothesis." While the former envisions an overt AI takeover pathway, characterized by scenarios like uncontrollable superintelligence, the latter suggests a different causal pathway to existential catastrophes. This involves a gradual accumulation of critical AI-induced threats such as severe vulnerabilities and systemic erosion of economic and political structures. The accumulative hypothesis suggests a boiling frog scenario where incremental AI risks slowly converge, undermining societal resilience until a triggering event results in irreversible collapse. Through systems analysis, this paper examines the distinct assumptions differentiating these two hypotheses. It is then argued that the accumulative view can reconcile seemingly incompatible perspectives on AI risks. The implications of differentiating between these causal pathways -- the decisive and the accumulative -- for the governance of AI as well as long-term AI safety are discussed.
Summary
Kasirzadeh challenges conventional AI existential risk thinking by proposing an 'accumulative AI x-risk hypothesis,' arguing catastrophe may arise gradually through interconnected disruptions—economic vulnerabilities, political erosion, systemic weaknesses—rather than abrupt superintelligence takeover. This 'boiling frog' framing offers a reconciliation between seemingly opposed perspectives on AI risk and carries distinct implications for governance and safety strategy.
Key Points
- •Contrasts 'decisive AI x-risk' (abrupt superintelligence takeover) with 'accumulative AI x-risk' (gradual erosion of societal resilience through compounding disruptions).
- •Argues existential catastrophe may result from incremental, interconnected failures crossing critical thresholds rather than a single decisive event.
- •Uses systems analysis to examine the distinct assumptions underlying each hypothesis and their different causal pathways.
- •The accumulative view can reconcile seemingly incompatible perspectives (e.g., near-term vs. long-term risk framings) within a unified framework.
- •Suggests different governance and long-term AI safety strategies are needed depending on which causal pathway is considered primary.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| AI-Induced Irreversibility | Risk | 64.0 |
Cached Content Preview
# Two Types of AI Existential Risk: Decisive and Accumulative
Atoosa Kasirzadeh
(atoosa.kasirzadeh@ed.ac.uk)
University of Edinburgh & Alan Turing Institute
###### Abstract
The conventional discourse on existential risks (x-risks) from AI typically focuses on abrupt, dire events caused by advanced AI systems, particularly those that might achieve or surpass human-level intelligence. These events have severe consequences that either lead to human extinction or irreversibly cripple human civilization to a point beyond recovery. This discourse, however, often neglects the serious possibility of AI x-risks manifesting incrementally through a series of smaller yet interconnected disruptions, gradually crossing critical thresholds over time. This paper contrasts the conventional _decisive AI x-risk hypothesis_ with an _accumulative AI x-risk hypothesis_. While the former envisions an overt AI takeover pathway, characterized by scenarios like uncontrollable superintelligence, the latter suggests a different causal pathway to existential catastrophes. This involves a gradual accumulation of critical AI-induced threats such as severe vulnerabilities and systemic erosion of econopolitical structures. The accumulative hypothesis suggests a boiling frog scenario where incremental AI risks slowly converge, undermining resilience until a triggering event results in irreversible collapse. Through systems analysis, this paper examines the distinct assumptions differentiating these two hypotheses. It is then argued that the accumulative view reconciles seemingly incompatible perspectives on AI risks. The implications of differentiating between these causal pathways — the decisive and the accumulative — for the governance of AI risks as well as long-term AI safety are discussed.
## 1 Introduction
Recent advances in machine learning have sparked intense debate about the existential risks (x-risks) associated with Artificial intelligence (AI) systems. Central to this debate is a concern about the potential pathways through which AI could cause existential catastrophes. In direct response to this concern, this paper aims to explore: How should we effectively conceptualize AI x-risks in the context of various types of causal pathways leading to AI-induced existential catastrophes? Conventional discourse on AI existential catastrophes typically portrays them as sudden, decisive events, often triggered by artificial general intelligence or artificial superintelligence (e.g., Bostrom ( [2013](https://ar5iv.labs.arxiv.org/html/2401.07836#bib.bib11 "")) and Ord ( [2020a](https://ar5iv.labs.arxiv.org/html/2401.07836#bib.bib38 ""))).
Contrasting this traditional decisive viewpoint, this paper introduces the _accumulative AI x-risk hypothesis_ as an alternative lens. The accumulative hypothesis posits that AI x-risks do not exclusively materialize as high-magnitude events initiated by artificial general intelligence or artificial superintelligence. Instead, they can
... (truncated, 78 KB total)987dd3f6f56b99cd | Stable ID: N2U4YTVmNj