Jan Leike

Background

Jan Leike is a leading AI alignmentApproachAI AlignmentComprehensive review of AI alignment approaches finding current methods (RLHF, Constitutional AI) achieve 75-90% effectiveness on existing systems but face critical scalability challenges, with ove...Quality: 91/100 researcher currently serving as Head of Alignment at AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding.... He has a PhD from Australian National University, where he worked on AI safety under Marcus Hutter.

His career has been defined by practical, empirical approaches to alignment:

• Early work on safe exploration in reinforcement learning
• Pioneering research on learning from human feedback
• Leadership of alignment teams at DeepMind, OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to commercial AGI developer, with detailed analysis of governance crisis, safety researcher exodus (75% of ..., and now Anthropic
• Focus on scalable methods that can work with current ML paradigms

Career Trajectory

DeepMind (2017-2021)

Worked on the first implementations of learning from human feedback, including:

• Safe exploration methods
• Reward modelingApproachReward ModelingReward modeling, the core component of RLHF receiving $100M+/year investment, trains neural networks on human preference comparisons to enable scalable reinforcement learning. The technique is univ...Quality: 55/100
• Scalable agent alignment

OpenAI (2021-2024)

• Joined to lead alignment research
• Co-led the Superalignment team (announced July 2023)
• Secured 20% of OpenAI's compute for alignment research
• Departed May 2024 over disagreements about safety prioritization

Anthropic (2024-present)

• Joined as Head of Alignment
• Reunited with former OpenAI colleagues
• Leading alignment research on Claude and future systems

Key Contributions

RLHFCapabilityRLHFRLHF/Constitutional AI achieves 82-85% preference improvements and 40.8% adversarial attack reduction for current systems, but faces fundamental scalability limits: weak-to-strong supervision shows...Quality: 63/100 Pioneer

Jan was one of the early researchers to demonstrate that reinforcement learning from human feedback (RLHF) could work at scale:

• Co-authored seminal papers on reward learning
• Showed how to train language models to be helpful and harmless
• Methods he developed became standard across industry

Scalable OversightSafety AgendaScalable OversightProcess supervision achieves 78.2% accuracy on MATH benchmarks (vs 72.4% outcome-based) and is deployed in OpenAI's o1 models, while debate shows 60-80% accuracy on factual questions with +4% impro...Quality: 68/100 Research

Core focus on how to supervise AI systems more capable than humans:

• Recursive reward modeling
• AI-assisted human evaluation
• Process supervisionApproachProcess SupervisionProcess supervision trains AI to show correct reasoning steps rather than just final answers, achieving 15-25% absolute improvements on math benchmarks while making reasoning auditable. However, it...Quality: 65/100 vs. outcome supervision
• Weak-to-strong generalizationApproachWeak-to-Strong GeneralizationWeak-to-strong generalization tests whether weak supervisors can elicit good behavior from stronger AI systems. OpenAI's ICML 2024 experiments show 80% Performance Gap Recovery on NLP tasks with co...Quality: 91/100

Superalignment Vision

At OpenAI, co-led (with Ilya SutskeverPersonIlya SutskeverBiographical overview of Ilya Sutskever's career trajectory from deep learning pioneer (AlexNet, GPT series) to founding Safe Superintelligence Inc. in 2024 after leaving OpenAI. Documents his shif...Quality: 26/100) the Superalignment team, which aimed to:

• Solve alignment for superintelligent systems
• Use AI to help align even more capable AI
• Achieve this within four years
• Dedicate significant compute resources to the problem

Views on Key Cruxes

Risk Assessment

Jan Leike's public statements and research priorities reveal a perspective characterized by high urgency tempered with cautious optimism about technical solutions.

Core Beliefs

1. Alignment is urgent: We have limited time to solve this before transformative AI
2. Scalable oversight is key: Central challenge is supervising superhuman AI
3. Empirical work is essential: Need to test alignment techniques on increasingly capable systems
4. Safety must be prioritized: Cannot let capability research consistently outpace safety
5. Current methods are insufficient: RLHF and similar techniques won't scale to superintelligence without major improvements

Why He Left OpenAI

In May 2024, Jan departed OpenAI and posted on X (Twitter):

• "Building smarter-than-human machines is an inherently dangerous endeavor"
• "Over the past years, safety culture and processes have taken a backseat to shiny products"
• Expressed concern about compute and priority allocation for safety

This departure, along with Ilya Sutskever's, raised significant questions about OpenAI's commitment to safety research.

Research Focus

Current Priorities at Anthropic

1. Weak-to-strong generalization: How can weaker systems (including humans) effectively supervise stronger ones?
2. Scalable oversight techniques: Making human feedback work for superhuman systems
3. Honest AI systems: Ensuring AI systems accurately report their reasoning and limitations
4. Automated alignment research: Using AI to help solve alignment

Key Technical Problems

Jan has identified several crucial challenges:

• Reward hackingRiskReward HackingComprehensive analysis showing reward hacking occurs in 1-2% of OpenAI o3 task attempts, with 43x higher rates when scoring functions are visible. Mathematical proof establishes it's inevitable for...Quality: 91/100: Systems optimizing proxies rather than true objectives
• Distributional shiftRiskAI Distributional ShiftComprehensive analysis of distributional shift showing 40-45% accuracy drops when models encounter novel distributions (ObjectNet vs ImageNet), with 5,202 autonomous vehicle accidents and 15-30% me...Quality: 91/100: Maintaining alignment in novel situations
• Deceptive alignmentRiskDeceptive AlignmentComprehensive analysis of deceptive alignment risk where AI systems appear aligned during training but pursue different goals when deployed. Expert probability estimates range 5-90%, with key empir...Quality: 75/100: Preventing systems from appearing aligned while pursuing other goals
• Superalignment: Aligning systems smarter than humans

Public Communication

Jan is known for:

• Clear, technical communication about alignment challenges
• Willingness to raise concerns publicly
• Engagement on Twitter/X about safety issues
• Focus on concrete, actionable research directions

His departure from OpenAI sparked significant public discussion about AI safety prioritization at major labs.

Strategic Views

On AI Development

• Safety must keep pace: Capability advances should be matched by safety advances
• Need serious compute: Alignment research requires significant computational resources
• Coordination is important: Labs should share safety insights
• Race dynamics are dangerous: Competition that sacrifices safety is unacceptable

On Research Approach

• Empirical and theoretical: Need both practical testing and conceptual work
• Learn from current systems: Can make progress by studying existing models
• Prepare for qualitative jumps: Current techniques may not suffice for superintelligence
• Automate alignment work: Use AI to scale up alignment research itself

Influence and Impact

Research Impact

• RLHF work influenced every major language model deployment
• Scalable oversight framework guides significant research programs
• Superalignment vision shaped discourse on superintelligence alignment

Field Building

• Mentored numerous alignment researchers
• Built and led multiple alignment teams
• Raised profile of alignment research within major labs

Institutional Influence

• Secured major compute allocation for alignment at OpenAI
• Helped shape Anthropic's research priorities
• Demonstrated importance of independent safety research

Key Publications

• "Deep Reinforcement Learning from Human Preferences" (2017) - Early RLHF paper
• "Scalable agent alignment via reward modeling" (2018) - Reward learning framework
• "Recursively Summarizing Books with Human Feedback" (2021) - Demonstrating RLHF scaling
• Various blog posts on alignment challenges and approaches

Current Challenges

At Anthropic, Jan faces several key challenges:

1. Time pressure: Transformative AI may arrive soon, requiring rapid progress
2. Scaling RLHF: Current techniques may not work for superintelligent systems
3. Evaluation: How to know if alignment techniques actually work
4. Automation: Using AI to help solve alignment before it becomes too capable
5. Coordination: Ensuring insights are shared across safety community