AI Safety Intervention Portfolio

Quick Assessment

Key Links

Overview

This page provides a strategic view of the AI safety intervention landscape, analyzing how different interventions address different risk categories. Rather than examining interventions individually, this portfolio view helps identify coverage gaps, complementarities, and allocation priorities.

The intervention landscape can be divided into several categories: technical approaches (alignment, interpretability, control), governance mechanisms (legislation, compute governance, international coordination), field building (talent, funding, community), and resilience measures (epistemic security, economic adaptation). Each category has different tractability profiles, timelines, and risk coverage—understanding these tradeoffs is essential for strategic resource allocation.

An effective safety portfolio requires both breadth (covering diverse failure modes) and depth (sufficient investment in each area to achieve impact). The current portfolio shows significant concentration in certain areas (RLHF, capability evaluations) while other areas remain relatively neglected (epistemic resilience, international coordination).

Field Growth Trajectory

Critical Comparison: While AI safety workforce has grown substantially, capabilities research is growing 30-40% per year. The ratio of capabilities to safety researchers has remained roughly constant at 10-15:1, meaning the absolute gap continues to widen.

Top Research Categories (by FTEs):

1. Miscellaneous technical AI safety researchCruxTechnical AI Safety ResearchTechnical AI safety research encompasses six major agendas (mechanistic interpretability, scalable oversight, AI control, evaluations, agent foundations, and robustness) with 500+ researchers and $...Quality: 66/100
2. LLM safety
3. Interpretability

Intervention Categories and Risk Coverage

flowchart TD
  subgraph Technical["Technical Approaches"]
      INT[Interpretability]
      CTRL[AI Control]
      ALIGN[Alignment Research]
      EVAL[Evaluations]
  end

  subgraph Governance["Governance"]
      COMP[Compute Governance]
      LEG[Legislation]
      INTL[International Coordination]
      RSP[Responsible Scaling]
  end

  subgraph Meta["Field Building & Resilience"]
      FIELD[Field Building]
      EPIST[Epistemic Resilience]
      ECON[Economic Resilience]
  end

  subgraph Risks["Risk Categories"]
      ACC[Accident Risks]
      MIS[Misuse Risks]
      STR[Structural Risks]
      EPI[Epistemic Risks]
  end

  INT --> ACC
  CTRL --> ACC
  ALIGN --> ACC
  EVAL --> ACC
  EVAL --> MIS

  COMP --> MIS
  COMP --> STR
  LEG --> MIS
  LEG --> STR
  INTL --> STR
  RSP --> ACC
  RSP --> MIS

  FIELD --> ACC
  FIELD --> STR
  EPIST --> EPI
  ECON --> STR

  style ACC fill:#ffcccc
  style MIS fill:#ffe6cc
  style STR fill:#fff3cc
  style EPI fill:#e6ccff
  style Technical fill:#cce6ff
  style Governance fill:#ccffcc
  style Meta fill:#ffccff

Intervention by Risk Matrix

This matrix shows how strongly each major intervention addresses each risk category. Ratings are based on current evidence and expert assessments.

Legend: High = primary focus, addresses directly; Medium = secondary impact; Low = indirect or limited; -- = minimal relevance

Prioritization Framework

This framework evaluates interventions across the standard Importance-Tractability-Neglectedness (ITN) dimensions, with additional consideration for timeline fit and portfolio complementarity.

Prioritization Rationale

Very High Priority:

• AI Control scores highly because it provides near-term safety benefits (70-85% tractability for human-level systems) regardless of whether alignment succeeds. It represents a practical bridge during the transition period. Redwood Research received $1.2M for control research in 2024.
• Compute Governance is one of few levers creating physical constraints on AI development. Hardware chokepoints exist, some measures are already implemented (EU AI Act compute thresholds, US export controls), and impact potential is substantial. GovAI produces leading research on compute governance mechanisms.

High Priority:

• Interpretability is potentially essential if alignment proves difficult (only reliable way to detect sophisticated deception). MIT Technology Review named mechanistic interpretability a 2026 Breakthrough Technology. Anthropic's attribution graphs revealed hidden reasoning in Claude 3.5 Haiku. FAS recommends federal R&D funding through DARPA and NSF.
• Evaluations provide measurable near-term impact and are already standard practice at major labs. Coefficient Giving launched an RFP for capability evaluations ($200K-$5M grants). METR partners with Anthropic and OpenAI on frontier model evaluations. NIST invested $20M in AI Economic Security Centers.
• International Coordination has very high impact potential for addressing structural risks like racing dynamics, but low tractability given current geopolitical tensions. The International AI Safety Report 2025, led by Yoshua BengioPersonYoshua BengioComprehensive biographical overview of Yoshua Bengio's transition from deep learning pioneer (Turing Award 2018) to AI safety advocate, documenting his 2020 pivot at Mila toward safety research, co...Quality: 39/100 with 100+ authors from 30 countries, represents the largest global collaboration to date.

Medium-High Priority:

• Field Building and Epistemic Resilience are relatively neglected meta-level interventions that multiply the effectiveness of direct technical and governance work. 80,000 Hours notes good funding opportunities in AI safety exist for qualified researchers.

Portfolio Gaps and Complementarities

Coverage Gaps

Analysis of the current intervention portfolio reveals several areas where coverage is thin:

Key Complementarities

Certain interventions work better together than in isolation:

Technical + Governance:

• AI EvaluationsResearch AreaAI EvaluationsEvaluations and red-teaming reduce detectable dangerous capabilities by 30-50x when combined with training interventions (o3 covert actions: 13% → 0.4%), but face fundamental limitations against so...Quality: 72/100 inform Responsible Scaling PoliciesApproachResponsible Scaling PoliciesComprehensive analysis of Responsible Scaling Policies showing 20 companies with published frameworks as of Dec 2025, with SaferAI grading major policies 1.9-2.2/5 for specificity. Evidence suggest...Quality: 62/100 thresholds
• InterpretabilityResearch AreaInterpretabilityMechanistic interpretability has extracted 34M+ interpretable features from Claude 3 Sonnet with 90% automated labeling accuracy and demonstrated 75-85% success in causal validation, though less th...Quality: 66/100 enables verification for
• AI ControlResearch AreaAI ControlAI Control is a defensive safety approach that maintains control over potentially misaligned AI through monitoring, containment, and redundancy, offering 40-60% catastrophic risk reduction if align...Quality: 75/100 provides safety margin while governance matures

Near-term + Long-term:

• Compute GovernanceConceptCompute GovernanceThis is a comprehensive overview of U.S. AI chip export controls policy, documenting the evolution from blanket restrictions to case-by-case licensing while highlighting significant enforcement cha...Quality: 58/100 buys time for InterpretabilityResearch AreaInterpretabilityMechanistic interpretability has extracted 34M+ interpretable features from Claude 3 Sonnet with 90% automated labeling accuracy and demonstrated 75-85% success in causal validation, though less th...Quality: 66/100 research
• AI EvaluationsResearch AreaAI EvaluationsEvaluations and red-teaming reduce detectable dangerous capabilities by 30-50x when combined with training interventions (o3 covert actions: 13% → 0.4%), but face fundamental limitations against so...Quality: 72/100 identify near-term risks while Scalable OversightResearch AreaScalable OversightProcess supervision achieves 78.2% accuracy on MATH benchmarks (vs 72.4% outcome-based) and is deployed in OpenAI's o1 models, while debate shows 60-80% accuracy on factual questions with +4% impro...Quality: 68/100 develops
• AI Safety Field Building and CommunityCruxAI Safety Field Building and CommunityGrowing the AI safety research community through funding, training, and outreach0 ensures capacity for future technical work

Prevention + Resilience:

• Technical safety research aims to prevent failures
• AI-Era Epistemic SecurityApproachAI-Era Epistemic SecurityComprehensive analysis of epistemic security finds human deepfake detection at near-chance levels (55.5%), AI detection dropping 45-50% on novel content, but content authentication (C2PA) market gr...Quality: 63/100 and economic resilience limit damage if prevention fails
• Both are needed for robust defense-in-depth

Portfolio Funding Allocation

The following table estimates 2024 funding levels by intervention area and compares them to recommended allocations based on neglectedness and impact potential. Total external AI safety funding was approximately $110-130 million in 2024, with Coefficient Giving providing ~60% of this amount.

2025 Funding Landscape Update

2025 Trajectory: Early data (through July 2025) shows $67M already committed, putting the year on track to exceed 2024 totals by 40-50%.

Funding Gap Analysis

The funding landscape reveals several structural imbalances:

Resource Allocation Assessment

Current vs. Recommended Allocation

Investment Concentration Risks

The current portfolio shows several structural vulnerabilities:

Strategic Considerations

Worldview Dependencies

Different beliefs about AI risk lead to different portfolio recommendations:

Portfolio Robustness

A robust portfolio should satisfy the following criteria, which can help evaluate current gaps and guide future allocation:

Key Sources