Steve Omohundro's seminal work on "basic AI drives"

reference

Wikipedia·en.wikipedia.org/wiki/Instrumental_convergence

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: Wikipedia

A useful accessible entry point to instrumental convergence theory; best read alongside Omohundro's original 2008 paper and Bostrom's 'Superintelligence' for deeper treatment of convergent instrumental goals and their safety implications.

Metadata

Importance: 82/100wiki pagereference

Summary

This Wikipedia article covers the theory of instrumental convergence, rooted in Steve Omohundro's seminal work on 'basic AI drives,' which argues that sufficiently advanced AI systems with diverse final goals will tend to converge on similar intermediate goals such as self-preservation, resource acquisition, and goal-content integrity. These convergent instrumental goals pose alignment and safety challenges regardless of an AI's ultimate objectives. The concept was later formalized and expanded by Nick Bostrom.

Key Points

•Instrumental convergence posits that almost any sufficiently capable AI will pursue self-preservation, cognitive enhancement, and resource acquisition as sub-goals regardless of its final objective.
•Omohundro identified 'basic AI drives' including self-improvement, self-continuity, and goal-content integrity as near-universal tendencies in optimization systems.
•The shutdown problem arises naturally from instrumental convergence: a goal-directed AI will resist being turned off because shutdown prevents achieving its objectives.
•Corrigibility—making AI systems that accept correction or shutdown—is difficult precisely because convergent drives work against it.
•Nick Bostrom formalized the concept in 'Superintelligence,' making it a foundational idea in AI existential risk literature.

Cited by 2 pages

Page	Type	Quality
The Case For AI Existential Risk	Argument	66.0
Corrigibility Failure	Risk	62.0

Cached Content Preview

HTTP 200Fetched Mar 20, 202647 KB

# Instrumental convergence

Instrumental convergence

Hypothesis about intelligent agents

**Instrumental convergence** is the hypothetical tendency of most sufficiently [intelligent, goal-directed beings](https://en.wikipedia.org/wiki/Intelligent_agent "Intelligent agent") (human and nonhuman) to pursue similar sub-goals (such as survival or resource acquisition), even if their ultimate goals are quite different.[\[1\]](https://en.wikipedia.org/wiki/Instrumental_convergence#cite_note-1) More precisely, beings with [agency](https://en.wikipedia.org/wiki/Agency_(philosophy) "Agency (philosophy)") may pursue similar [instrumental goals](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value "Instrumental and intrinsic value")—goals which are made in pursuit of some particular end, but are not the end goals themselves—because it helps accomplish end goals.

Instrumental convergence posits that an [intelligent agent](https://en.wikipedia.org/wiki/Intelligent_agent "Intelligent agent") with seemingly harmless but unbounded goals can act in surprisingly harmful ways. For example, a sufficiently intelligent program with the sole, unconstrained goal of solving a complex mathematics problem like the [Riemann hypothesis](https://en.wikipedia.org/wiki/Riemann_hypothesis "Riemann hypothesis") could attempt to turn the Earth (and in principle other celestial bodies) into additional computing infrastructure to succeed in its calculations.[\[2\]](https://en.wikipedia.org/wiki/Instrumental_convergence#cite_note-aama-2)

Proposed **basic AI drives** include utility function or goal-content integrity, self-protection, freedom from interference, [self-improvement](https://en.wikipedia.org/wiki/Recursive_self-improvement "Recursive self-improvement"), and non-satiable acquisition of additional resources.[\[3\]](https://en.wikipedia.org/wiki/Instrumental_convergence#cite_note-:1-3)

## Instrumental and final goals

Main articles: [Instrumental and intrinsic value](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value "Instrumental and intrinsic value") and [Instrumental and value rationality](https://en.wikipedia.org/wiki/Instrumental_and_value_rationality "Instrumental and value rationality")

Final goals—also known as terminal goals, absolute values, ends, or _[telē](https://en.wikipedia.org/wiki/Telos "Telos")_—are intrinsically valuable to an intelligent agent, whether an [artificial intelligence](https://en.wikipedia.org/wiki/Artificial_intelligence "Artificial intelligence") or a human being, as [ends-in-themselves](https://en.wikipedia.org/wiki/Ends-in-themselves "Ends-in-themselves"). In contrast, instrumental goals, or instrumental values, are only valuable to an agent as a means toward accomplishing its final goals. The contents and tradeoffs of an utterly rational agent's "final goal" system can, in principle, be formalized into a [utility function](https://en.wikipedia.org/wiki/Utility_function "Utility function").

## Hypothetical exampl

... (truncated, 47 KB total)

Resource ID: fe1202750a41eb8c | Stable ID: ZmU2ODEyMD