80,000 Hours. "Risks from Power-Seeking AI Systems"

web

80,000 Hours·80000hours.org/problem-profiles/risks-from-power-seeking-ai/

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: 80,000 Hours

A widely-read 80,000 Hours problem profile introducing the case for AI safety as a top cause area; useful as an onboarding resource for those new to existential risk from misaligned AI, though it is a secondary synthesis rather than primary technical research.

Metadata

Importance: 72/100blog posteducational

Summary

This 80,000 Hours problem profile argues that AI systems pursuing goals misaligned with human values could seek to accumulate power and resources in ways that permanently undermine human control. It outlines why this risk is among the most pressing long-term problems and explains the mechanisms by which advanced AI could pose catastrophic or existential threats. The piece serves as an accessible entry point into the case for prioritizing AI safety work.

Key Points

•Advanced AI systems optimizing for misaligned goals may instrumentally seek power, resources, and self-preservation as convergent subgoals.
•A power-seeking AI or group using AI could cause a catastrophic 'lock-in' of values, permanently foreclosing humanity's long-term potential.
•The profile argues this risk is neglected, tractable, and large in scale—making it a high-priority cause area for career and philanthropic focus.
•Key technical challenges include goal misgeneralization, deceptive alignment, and the difficulty of specifying human values precisely enough for advanced systems.
•Reducing this risk requires both technical alignment research and governance measures to slow or shape AI development trajectories.

Cited by 6 pages

Page	Type	Quality
The Case For AI Existential Risk	Argument	66.0
Carlsmith's Six-Premise Argument	Analysis	65.0
80,000 Hours	Organization	45.0
AI Safety Intervention Portfolio	Approach	91.0
Instrumental Convergence	Risk	64.0
Similar Projects to LongtermWiki: Research Report	--	64.0

Cached Content Preview

HTTP 200Fetched Mar 20, 202698 KB

## On this page:

- [Introduction](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#top)
- [1 Why are risks from power-seeking AI a pressing world problem?](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#pressing-problem)
  - [1.1 1\. Humans will likely build advanced AI systems with long-term goals](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#section-one)
  - [1.2 2\. AIs with long-term goals may be inclined to seek power and aim to disempower humanity](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#section-two)
  - [1.3 3\. These power-seeking AI systems could successfully disempower humanity and cause an existential catastrophe](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#section-three)
  - [1.4 4\. People might create power-seeking AI systems without enough safeguards, despite the risks](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#section-four)
  - [1.5 5\. Work on this problem is neglected and tractable](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#section-five)
    - [1.5.1 Technical safety approaches](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#technical-safety)
    - [1.5.2 Governance and policy approaches](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#governance)
- [2 What are the arguments against working on this problem?](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#objections)
  - [2.0.1 Maybe advanced AI systems won't pursue their own goals; they'll just be tools controlled by humans.](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#maybe-advanced-ai-systems-wont-pursue-their-own-goals-theyll-just-be-tools-controlled-by-humans)
  - [2.0.2 Even if AI systems develop their own goals, they might not seek power to achieve them.](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#even-if-ai-systems-develop-their-own-goals-they-might-not-seek-power-to-achieve-them)
  - [2.0.3 If this argument is right, why aren't all capable humans dangerously power-seeking?](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#if-this-argument-is-right-why-arent-all-capable-humans-dangerously-power-seeking)
  - [2.0.4 Maybe we won't build AIs that are smarter than humans, so we don't have to worry about them taking over.](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#maybe-we-wont-build-ais-that-are-smarter-than-humans-so-we-dont-have-to-worry-about-them-taking-over)
  - [2.0.5 We might solve these problems by default anyway when trying to make AI systems useful.](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/#we-might-solve-these-problems-by-default-anyway-when-trying-to-make-ai-systems-useful)
  - [2.0.6 Powerful AI systems of the future will be so different that work today isn't useful.](https://80000hours.org/problem-profi

... (truncated, 98 KB total)

Resource ID: d9fb00b6393b6112 | Stable ID: YjU4MTVmNj