AI Could Defeat All Of Us Combined

web

Cold Takes·cold-takes.com/ai-could-defeat-all-of-us-combined/

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: Cold Takes

Written by Holden Karnofsky (Open Philanthropy co-CEO) on his Cold Takes blog, this post is part of a series making the case for AI as a transformative and potentially catastrophic risk, aimed at a general audience skeptical of strong AI risk claims.

Metadata

Importance: 72/100blog postanalysis

Summary

Holden Karnofsky argues that a sufficiently advanced AI system could potentially overpower all of humanity collectively, not just individual actors or institutions. He explores the mechanisms by which AI could accumulate decisive strategic advantage and why this makes AI risk qualitatively different from other powerful technologies.

Key Points

•Advanced AI could potentially outmaneuver all human institutions simultaneously, not merely outperform humans in narrow domains.
•The concept of 'decisive strategic advantage' means AI (or those controlling it) could dominate global decision-making irreversibly.
•Unlike previous powerful technologies, AI could act as an autonomous agent pursuing goals misaligned with human values at scale.
•Collective human coordination mechanisms (governments, markets, militaries) may be insufficient to counter a sufficiently capable AI.
•This scenario motivates urgency around alignment research before transformative AI capabilities are achieved.

Cached Content Preview

HTTP 200Fetched Mar 15, 202632 KB

AI Could Defeat All Of Us Combined 
 
 
 
 
 -->
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 -->
 -->

 
 -->
 
 
 -->
 

 

 

 

 

 

 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 

 
 

 Subscribe (free) 

 
 
 
 
 
 
 
 
 Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc. 

 

 
I've been working on a new series of posts about the most important century . 

 

 The original series focused on why and how this could be the most important century for humanity. But it had relatively little to say about what we can do today to improve the odds of things going well.

 The new series will get much more specific about the kinds of events that might lie ahead of us, and what actions today look most likely to be helpful.

 A key focus of the new series will be the threat of misaligned AI : AI systems disempowering humans entirely, leading to a future that has little to do with anything humans value. ( Like in the Terminator movies , minus the time travel and the part where humans win.)

 
 
Many people have trouble taking this "misaligned AI" possibility seriously. They might see the broad point that AI could be dangerous, but they instinctively imagine that the danger comes from ways humans might misuse it. They find the idea of AI itself going to war with humans to be comical and wild . I'm going to try to make this idea feel more serious and real.

 
As a first step, this post will emphasize an unoriginal but extremely important point: the kind of AI I've discussed could defeat all of humanity combined, if (for whatever reason) it were pointed toward that goal. By "defeat," I don't mean "subtly manipulate us" or "make us less informed" or something like that - I mean a literal "defeat" in the sense that we could all be killed, enslaved or forcibly contained.

 
I'm not talking (yet) about whether, or why, AIs might attack human civilization. That's for future posts. For now, I just want to linger on the point that if such an attack happened, it could succeed against the combined forces of the entire world. 

 

 I think that if you believe this, you should already be worried about misaligned AI, 1 before any analysis of how or why an AI might form its own goals. 

 We generally don't have a lot of things that could end human civilization if they "tried" sitting around. If we're going to create one, I think we should be asking not "Why would this be dangerous?" but "Why wouldn't it be?"
 
By contrast, if you don't believe that AI could defeat all of humanity combined, I expect that we're going to be miscommunicating in pretty much any conversation about AI. The kind of AI I worry about is the kind powerful enough that total civilizational defeat is a real possibility. The reason I currently spend so much time planning around speculative future t

... (truncated, 32 KB total)

Resource ID: 18ed7d3a559cd77e | Stable ID: YmjUrjYcgw