Skip to content
Longterm Wiki has not been actively maintained since May 2026. Some data is still updated occasionally, but please don't rely on this site being current or accurate.
Longterm Wiki

Credibility Rating

2/5
Mixed(2)

Mixed quality. Some useful content but inconsistent editorial standards. Claims should be verified.

Rating inherited from publication venue: Medium

Official update from Google DeepMind's main technical safety team, useful for understanding the institutional structure and research priorities of one of the largest industry AI safety groups as of late 2024.

Metadata

Importance: 62/100blog postprimary source

Summary

A 2024 overview by Google DeepMind's AGI Safety & Alignment team summarizing their recent technical work on existential risk from AI, covering subteams focused on mechanistic interpretability, scalable oversight, and frontier safety evaluations. Written by Rohin Shah, Seb Farquhar, and Anca Dragan, it describes the team's structure, growth, and key research priorities including amplified oversight and dangerous capability evaluations.

Key Points

  • •The AGI Safety & Alignment team has grown ~39% and ~37% in consecutive years, encompassing subteams in mechanistic interpretability, scalable oversight, and frontier safety.
  • •Big bets for 2023-2024 include amplified oversight (to generate correct learning signals for alignment) and other technical safety approaches.
  • •The Frontier Safety subteam develops and runs dangerous capability evaluations as part of DeepMind's Frontier Safety Framework.
  • •The broader org also includes Gemini Safety (safety training for current models) and Voices of All in Alignment (value/viewpoint pluralism techniques).
  • •The post is intended to help external researchers build on DeepMind's work and understand how their own research connects to this agenda.

Cited by 3 pages

Cached Content Preview

HTTP 200Fetched Nov 3, 202618 KB
AGI Safety and Alignment at Google DeepMind: A Summary of Recent Work

 DeepMind Safety Research 11 min read · Oct 18, 2024 -- 

 Listen

 Share

 By Rohin Shah, Seb Farquhar, and Anca Dragan 

 It’s been a while since our last update, so we wanted to share a recap of our recent outputs. Below, we fill in some details about what we have been working on, what motivated us to do it, and how we thought about its importance. We hope that this will help people build off things we have done and see how their work fits with ours.

 Who are we?

 We’re the main team at Google DeepMind working on technical approaches to existential risk from AI systems. Since our last post , we’ve evolved into the AGI Safety & Alignment team, which we think of as AGI Alignment (with subteams like mechanistic interpretability, scalable oversight, etc.), and Frontier Safety (working on the Frontier Safety Framework , including developing and running dangerous capability evaluations). We’ve also been growing since our last post: by 39% last year, and by 37% so far this year. The leadership team is Anca Dragan, Rohin Shah, Allan Dafoe, and Dave Orr, with Shane Legg as executive sponsor. We’re part of the overall AI Safety and Alignment org led by Anca, which also includes Gemini Safety (focusing on safety training for the current Gemini models), and Voices of All in Alignment, which focuses on alignment techniques for value and viewpoint pluralism.

 What have we been up to?

 Here is some key work published in 2023 and the first part of 2024, grouped by topic / sub-team.

 Our big bets for the past 1.5 years have been 1) amplified oversight, to enable the right learning signal for aligning models so that they don’t pose catastrophic risks, 2) frontier safety, to analyze whether models are capable of posing catastrophic risks in the first place, and 3) (mechanistic) interpretability, as a potential enabler for both frontier safety and alignment goals. Beyond these bets, we experimented with promising areas and ideas that help us identify new bets we should make.

 Frontier Safety

 The mission of the Frontier Safety team is to ensure safety from extreme harms by anticipating, evaluating, and helping Google prepare for powerful capabilities in frontier models. While the focus so far has been primarily around misuse threat models, we are also working on misalignment threat models.

 FSF

 We recently published our Frontier Safety Framework , which, in broad strokes, follows the approach of responsible capability scaling , similar to Anthropic’s Responsible Scaling Policy and OpenAI’s Preparedness Framework . The key difference is that the FSF applies to Google: there are many different frontier LLM deployments across Google, rather than just a single chatbot and API (this in turn affects stakeholder engagement, policy implementation, mitigation plans, etc).

 We’re excited that our small team led the Google-wide strategy in this space, and demonstrated that responsible capabili

... (truncated, 18 KB total)
Resource ID: 6374381b5ec386d1 | Stable ID: sid_YVGTVch0VX