Skip to content
Longterm Wiki
Back

AGI Ruin: A List of Lethalities

web
2022·LessWrong·lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-...

Author

Eliezer Yudkowsky

Credibility Rating

3/5
Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: LessWrong

Yudkowsky's comprehensive catalog of technical and strategic reasons why AGI alignment is likely to fail, written as a direct rebuttal to optimistic AI safety narratives. Highly influential in the AI safety community as a statement of the 'doom' position.

Forum Post Details

Karma
970
Comments
713
Forum
lesswrong
Status
Curated
Forum Tags
AI RiskThreat Models (AI)AI Questions Open ThreadsDouble-CruxFuzziesLanguage Models (LLMs)Meetups & Local Communities (topic)AI
Part of sequence: 2022 MIRI Alignment Discussion

Metadata

Importance: 90/100blog postanalysis

Summary

Eliezer Yudkowsky catalogs the specific technical and practical reasons why aligning AGI systems is likely to fail catastrophically. The post argues that the minimal bar—AGI that won't kill everyone—is not being met by current approaches, and systematically addresses why common proposed solutions are insufficient. It covers foundational concepts like orthogonality and instrumental convergence before detailing specific alignment failure modes.

Key Points

  • •The minimal alignment standard is not perfect safety but simply AGI that won't kill everyone—a bar Yudkowsky argues current approaches fail to meet.
  • •Orthogonality and instrumental convergence imply that capable AGI systems will by default pursue goals misaligned with human survival.
  • •Current ML training paradigms produce systems whose internal goals are opaque and not reliably shaped by the training objective.
  • •Mesa-optimization and inner alignment failures mean that even well-specified training objectives may produce deceptively aligned models.
  • •The post argues that the AI safety field is not on a trajectory to solve these problems before transformative AGI is developed.

Cited by 1 page

PageTypeQuality
Deceptive AlignmentRisk75.0

Cached Content Preview

HTTP 200Fetched Mar 30, 2026557 KB
x This website requires javascript to properly function. Consider activating javascript to get access to all site functionality. AGI Ruin: A List of Lethalities — LessWrong window.__lwSsrGql.inject("query postCommentsThreadQuery($selector: CommentSelector, $limit: Int, $enableTotal: Boolean) {\n comments(selector: $selector, limit: $limit, enableTotal: $enableTotal) {\n results {\n ...CommentsList\n }\n totalCount\n }\n}\n\nfragment TagBasicInfo on Tag {\n _id\n userId\n name\n shortName\n slug\n core\n postCount\n adminOnly\n canEditUserIds\n suggestedAsFilter\n needsReview\n descriptionTruncationCount\n createdAt\n wikiOnly\n deleted\n isSubforum\n noindex\n isArbitalImport\n isPlaceholderPage\n baseScore\n extendedScore\n score\n afBaseScore\n afExtendedScore\n voteCount\n currentUserVote\n currentUserExtendedVote\n}\n\nfragment TagPreviewFragment on Tag {\n ...TagBasicInfo\n isRead\n parentTag {\n ...TagBasicInfo\n }\n subTags {\n ...TagBasicInfo\n }\n description {\n _id\n htmlHighlight\n }\n canVoteOnRels\n authorOnly\n isArbitalImport\n}\n\nfragment UsersMinimumInfo on User {\n _id\n slug\n createdAt\n username\n displayName\n profileImageId\n karma\n afKarma\n deleted\n isAdmin\n htmlBio\n postCount\n commentCount\n sequenceCount\n afPostCount\n afCommentCount\n spamRiskScore\n tagRevisionCount\n reviewedByUserId\n}\n\nfragment CommentsList on Comment {\n _id\n postId\n tagId\n tag {\n _id\n slug\n }\n relevantTagIds\n relevantTags {\n ...TagPreviewFragment\n }\n tagCommentType\n parentCommentId\n topLevelCommentId\n descendentCount\n title\n contents {\n _id\n html\n plaintextMainText\n wordCount\n }\n postedAt\n lastEditedAt\n repliesBlockedUntil\n userId\n draft\n deleted\n deletedPublic\n deletedByUserId\n deletedReason\n hideAuthor\n authorIsUnreviewed\n user {\n ...UsersMinimumInfo\n }\n currentUserVote\n currentUserExtendedVote\n isBookmarked\n baseScore\n extendedScore\n score\n voteCount\n af\n afDate\n moveToAlignmentUserId\n afBaseScore\n afExtendedScore\n suggestForAlignmentUserIds\n reviewForAlignmentUserId\n needsReview\n answer\n parentAnswerId\n retracted\n postVersion\n reviewedByUserId\n shortform\n shortformFrontpage\n lastSubthreadActivity\n moderatorHat\n hideModeratorHat\n nominatedForReview\n reviewingForReview\n promoted\n promotedByUser {\n ...UsersMinimumInfo\n }\n directChildrenCount\n votingSystem\n isPinnedOnProfile\n debateResponse\n rejected\n rejectedReason\n originalDialogueId\n}\n::\n{\"enableTotal\":false,\"limit\":1000,\"selector\":{\"postCommentsTop\":{\"postId\":\"uMQ3cqWDPHhjtiesc\"}}}",{"data":{"comments":{"results":[{"$ref":"zhkEnmChwaiW5eXx8"},{"$ref":"u3MgyyP3TDijmGhec"},{"$ref":"HRDoDnHv8bvoW7oPZ"},{"$ref":"m3ALfcpXGnPkh9A5n"},{"$ref":"9ZhXbv8p2fr8mkXaa"},{"$ref":"4nSEcid6bboaNQdxm"},{"$ref":"AhnLhgAQkFbciQFfM"},{"$ref":"pyz8NJonktCexRjuh"},{"$ref":"JL29PYktEvpd6fuQ5"},{"$ref":"SG8KaKeZbnBTEWrYA"},{"$ref":"LowEED2iDkhco3a5d"},{"$ref":"ePAXXk8AvpdGeynHe"},{"$ref":"wMDw8Ce3YqtcvXpry"},{"$ref":"Kvv

... (truncated, 557 KB total)
Resource ID: ebf69d1a871a8145 | Stable ID: YmRlOWY0ZD