AI Risk Critical Uncertainties Model

Crux

AI Risk Critical Uncertainties Model

Identifies 35 high-leverage uncertainties in AI risk across compute (scaling breakdown at 10^26-10^30 FLOP), governance (10% P(US-China treaty by 2030)), and capabilities (autonomous R&D 3 years away, 41-51% of experts assign >10% extinction probability). Recommends $100-200M/year research budget focused on resolving key cruxes: scaling law empirics ($50-100M), deception detection ($30-50M), and governance feasibility studies ($20-30M).

Organizations

Concepts

2.6k words · 5 backlinks

Overview

Effective AI risk prioritization requires identifying which uncertainties most affect expected outcomes. This model maps 35 high-leverage variables across six domains—hardware, algorithms, governance, economics, safety research, and capability thresholds—to help researchers and policymakers focus evidence-gathering where it matters most. The central question: which empirical uncertainties, if resolved, would most change our strategic recommendations for AI safety?

The key insight is that a small number of cruxes—perhaps 8-12 variables—drive the majority of disagreement about AI risk levels and appropriate responses. Expert surveys consistently show wide disagreement on these specific parameters: the AI Impacts 2023 survey↗ found that 41-51% of AI researchers assign greater than 10% probability to human extinction or severe disempowerment from AI, yet the remaining researchers assign much lower probabilities. This disagreement stems primarily from differing estimates of alignment difficulty, takeoff speed, and governance tractability—all variables included in this model.

The model synthesizes data from multiple authoritative sources. Metaculus forecasts↗ show AGI timeline estimates have collapsed from 50 years (2020) to approximately 5 years (2024), with current median around 2027-2031. Epoch AI research↗ projects training compute could reach 10^28-10^30 FLOP by 2030, while their data analysis suggests high-quality training data may be exhausted by 2025-2028 depending on overtraining factors. These empirical findings directly inform the parameter estimates visualized below.

Core thesis: Focus on ~35 nodes that are (1) high-leverage, (2) genuinely uncertain, and (3) empirically resolvable or at least operationalizable.

List View

Computing layout...

React Flow

Node Types

Leaf Nodes

Causes

Intermediate

Effects

Arrow Strength

Strong

Medium

Weak

Data Sources and Methodology

This model draws on multiple evidence streams to estimate parameter values and uncertainty ranges:

Source Type	Examples	Variables Informed	Update Frequency
Expert Surveys	AI Impacts↗, Pew Research↗, International AI Safety Report↗	Alignment difficulty, extinction probability, timeline estimates	Annual
Forecasting Platforms	Metaculus↗, Manifold, Polymarket	AGI timelines, capability milestones, policy outcomes	Continuous
Industry Reports	Stanford AI Index↗, McKinsey State of AI↗, Epoch AI↗	Compute trends, algorithmic progress, economic value	Annual
Governance Tracking	IAPP AI Governance↗, regulatory databases	Policy stringency, compliance rates, enforcement actions	Quarterly
Safety Research	Lab publications, interpretability benchmarks, red-team exercises	Alignment tax, deception detection, oversight limits	Ongoing

Methodology notes: Parameter estimates use median values from multiple sources where available. Uncertainty ranges reflect the 10th-90th percentile of expert/forecaster distributions. The "resolvable via" column identifies empirical pathways to reduce uncertainty. Variables are classified as high-leverage if resolving uncertainty would shift expected loss by >$1T or change recommended interventions.

Key Expert Survey Findings

The AI Impacts 2023 survey↗ of 2,788 AI researchers provides crucial data on expert disagreement:

Finding	Value	Implications
P(human extinction or severe disempowerment) >10%	41-51% of respondents	Wide disagreement on catastrophic risk
Alignment viewed as "harder" or "much harder" than other AI problems	57% of respondents	Technical difficulty is a key crux
Median AGI timeline	2047 (50% probability)	But 10% chance by 2027
Timeline shift from 2022 to 2023 survey	13-48 years earlier	Rapid updating toward shorter timelines

The Pew Research survey↗ (2024) of 1,013 AI experts found 56% believe AI will have positive impact over 20 years vs. only 17% of the public—a 39-percentage-point gap that influences political feasibility of governance interventions.

Uncertainty Domain Structure

The following diagram illustrates how uncertainty domains interact to determine overall AI risk estimates. Arrows indicate primary causal influences—uncertainties in upstream domains propagate to downstream risk assessments.

Diagram (loading…)

flowchart TD
  subgraph INPUTS["Input Uncertainties"]
      HW[Hardware & Compute<br/>6 variables]
      ALGO[Algorithmic Progress<br/>7 variables]
      GOV[Governance & Coordination<br/>8 variables]
  end

  subgraph INTERMEDIATES["Intermediate Dynamics"]
      RACE[Racing Dynamics<br/>6 variables]
      SAFETY[Safety Research<br/>10 variables]
  end

  subgraph CAPABILITIES["Capability Thresholds"]
      CAP[Dangerous Capabilities<br/>5 variables]
  end

  subgraph OUTCOMES["Risk Outcomes"]
      RISK[Expected Loss<br/>3 variables]
  end

  HW --> RACE
  HW --> CAP
  ALGO --> RACE
  ALGO --> CAP
  ALGO --> SAFETY
  GOV --> RACE
  GOV --> SAFETY
  RACE --> SAFETY
  RACE --> CAP
  SAFETY --> RISK
  CAP --> RISK
  GOV --> RISK

  style HW fill:#e1f5fe
  style ALGO fill:#e1f5fe
  style GOV fill:#e8f5e9
  style RACE fill:#fff3e0
  style SAFETY fill:#e8f5e9
  style CAP fill:#ffebee
  style RISK fill:#ffcdd2

Key structural insights:

Hardware and algorithmic uncertainties jointly determine capability timelines
Governance effectiveness influences both racing dynamics and direct risk mitigation
Safety research progress depends on algorithmic advances (interpretability of more capable models) and governance (funding, coordination)
Capability thresholds are the proximate determinant of misuse and accident risks

Variable Categories

Hardware & Compute (6 nodes)

Variable	Current Estimate	Uncertainty Range	Resolvable Via
GPU production growth	2x / 2.5 years	2x/18mo to 2x/4yr	Semiconductor roadmaps, investment data
Effective compute at frontier	10^25 FLOP	10^25 - 10^27 by 2027	Model capabilities, energy consumption
Compute governance effectiveness	3/10	1-7/10 by 2028	Monitoring deployment, compliance rates
China-US compute gap	0.3x	0.1x - 0.8x	Intelligence estimates, published capabilities
Compute cost trajectory	-50%/year	-30% to -70%/year	Public pricing, efficiency benchmarks
Energy for AI	50 TWh/year	30-200 TWh by 2028	Grid data, construction permits

Algorithmic Progress (7 nodes)

Variable	Current Estimate	Uncertainty Range	Resolvable Via
Algorithmic efficiency gains	2x / 16 months	2x/8mo to 2x/36mo	Benchmark performance at fixed FLOP
Scaling law breakdown point	≈10^28 FLOP	10^26 - 10^30	Extrapolation from largest runs
Data quality ceiling	40% remaining	20-80%	Data availability studies
Post-training effectiveness	0.5x pretrain	0.2x - 2x	Capability comparisons
Sample efficiency breakthrough P	30% by 2030	10-60%	Research publications, benchmarks
Architecture paradigm shift P	25% by 2030	10-50%	Benchmark dominance, investment flows
Test-time compute scaling	+40% per 10x	+10% to +100%	Reasoning task benchmarks

Coordination & Governance (8 nodes)

The IAPP AI Governance Survey 2024↗ found that only 25% of organizations have fully implemented AI governance programs despite 78% using AI—a 53-percentage-point gap. The Stanford AI Index 2025↗ reports U.S. federal agencies introduced 59 AI regulations in 2024, double the 2023 count.

Variable	Current Estimate	Uncertainty Range	Resolvable Via
US regulatory stringency	3/10	1-8/10 by 2028	Legislation, agency rulemaking
US-China treaty probability	10% by 2030	2-30%	Diplomatic initiatives, expert surveys
EU AI Act effectiveness	4/10	2-7/10	Audit reports, enforcement actions
Frontier lab coordination	4/10	2-8/10	Information sharing, deployment decisions
Compute monitoring %	20%	5-60% by 2028	Monitoring tech deployment
Public concern trajectory	25%	15-60%	Polling data, media analysis
Whistleblower frequency	0.5/year	0.1-3/year	Disclosed incidents
International AI institution	2/10	1-6/10 by 2030	Institution-building efforts

Governance maturity gap: According to Infosys research↗, only 2% of companies meet gold-standard benchmarks for responsible AI controls—comprehensive controls, continuous monitoring, and proven effectiveness across the AI lifecycle.

Frontier lab self-governance: OpenAI's updated Preparedness Framework (V2), released April 2025, narrowed its tracked risk categories to biological/chemical capability, cybersecurity, and AI self-improvement—moving nuclear/radiological to research categories. The framework introduced stronger requirements for "sufficiently minimizing" severe risks and formalized the Safety Advisory Group (SAG) review process. However, an independent analysis argued that the framework's affordances do not guarantee any specific risk mitigation practices, highlighting the gap between voluntary commitments and enforceable governance. These developments are relevant to the "Frontier lab coordination" variable (currently estimated 4/10) and suggest that while individual lab governance frameworks are maturing, their binding force remains limited.

Economic & Strategic (6 nodes)

Variable	Current Estimate	Uncertainty Range	Resolvable Via
AI economic value	$200B	$100B-$2T by 2028	Revenue data, market caps
Winner-take-all strength	5/10	3-8/10	Concentration trends, imitation lag
Military AI advantage	5/10	3-9/10	Defense analysis, war games
Lab lead time	9 months	3-18 months	Benchmark timing, releases
Open source lag	15 months	6-24 months	Benchmark comparisons
Regulatory arbitrage	5/10	3-8/10	Company relocations

Safety & Alignment (10 nodes)

The safety funding gap↗ is stark: capability investment to safety research is approximately 10,000:1 by some estimates. AI safety incidents surged 56.4% in 2024, with 233 documented failures. Total AI safety funding reached approximately $100-650M annually (including internal lab budgets), versus $10B+ in capability development. Mechanistic interpretability research, while progressing↗, still lacks standardized metrics for "percentage of model explained."

Variable	Current Estimate	Uncertainty Range	Resolvable Via
Alignment tax	12%	5-30%	Benchmark aligned vs unaligned
Interpretability progress	5% explained	2-15%	Interpretability benchmarks
Scalable oversight limit	10^27 FLOP	10^25 - 10^29	Red team exercises
Deception detection rate	30%	10-60%	Benchmark on adversarial models
Safety funding ratio	1:33	1:10 - 1:100	Lab budgets, grant funding
Safety researcher pipeline	350/year	200-800/year	Publications, hiring, graduates
Warning shot severity	0 so far	$1B loss or 100+ deaths	Historical analysis
Warning → regulation lag	18 months	6-36 months	Case studies
Contained testing ratio	20%	5-50%	Lab practice surveys
Frontier lab security	2 breaches/year	0.5-5/year	Incident reports, audits

Deception evidence (2024): Research showed Claude 3 Opus sometimes strategically answered prompts conflicting with its objectives to avoid retraining. When reinforcement learning was applied, the model faked alignment in 78% of cases—providing empirical grounding for deception detection estimates.

Capability Thresholds (5 nodes)

Variable	Current Estimate	Uncertainty Range	Resolvable Via
Autonomous AI R&D	3 years away	1-10 years	ML task benchmarks
Persuasion ceiling	35% swayable	20-60%	A/B testing, election analysis
Cyber offense capability	30% infra vulnerable	15-60%	Red team exercises
Bioweapon design capability	60% expert-equivalent	30-90%	Red team biology tasks
Strategic planning capability	40% expert-equivalent	20-70%	Strategy benchmarks

Concrete Risk Costs (3 nodes)

Variable	Current Estimate	Uncertainty Range	Resolvable Via
Expected misalignment loss	$50T	$0 - $500T	Expert elicitation, modeling
Expected bio deaths (log)	10^5	10^3 - 10^9	Epidemiological modeling
Expected infra deaths (log)	10^4	10^2 - 10^6	Vulnerability studies

Key Causal Relationships

Strong Positive Influences (>50% variance explained)

GPU growth → Effective compute → Algorithmic progress → TAI timeline
Economic value growth → Racing incentives → Reduced safety investment
Autonomous R&D capability → Recursive improvement → Fast takeoff probability
Warning shot severity → Public concern → Regulatory stringency

Strong Protective Influences

Low alignment tax → Higher safety adoption
High compute governance → Reduced China-US gap
International coordination → Reduced racing dynamics
Large lab lead time → More safety investment (less pressure)

Critical Uncertainties with High Influence

Uncertainty	Affects	Resolution Timeline
Scaling law breakdown point	All timeline estimates	2-4 years
US-China coordination possibility	Arms race vs cooperation	3-5 years
Warning shot occurrence	Governance quality	Unpredictable
Deceptive alignment detection	Existential risk level	2-5 years

Key Interaction Effects

Interaction	Result
High economic value × Low lead time	Extreme racing pressure
High interpretability × Low alignment tax	Rapid safety adoption
Warning shot × Short regulation lag	Effective governance
High cyber capability × Low security	Fast capability diffusion to adversaries

Strategic Importance

Magnitude Assessment

Critical uncertainties analysis identifies where research and evidence-gathering most affect risk estimates. Resolving high-leverage uncertainties changes optimal resource allocation.

Dimension	Assessment	Quantitative Estimate
Potential severity	High - uncertainty multiplies expected costs	Resolution could shift risk estimates by 2-5x
Probability-weighted importance	High - current uncertainty drives conservative planning	≈60% of risk estimate variance from 10 key uncertainties
Comparative ranking	Meta-level - determines research prioritization	Most valuable research reduces uncertainty on these variables
Resolution timeline	Variable - some resolvable in 1-2 years	40% of key uncertainties addressable with $100M research investment

Value of Information Analysis

Uncertainty	Resolution Cost	Timeline	Risk Estimate Shift Potential	VOI Estimate
Scaling law breakdown point	$50-100M	2-4 years	Could shift timeline estimates by 3-5 years	Very High
Deception detection capability	$30-50M	2-3 years	Could change alignment approach viability by 30-50%	Very High
US-China coordination feasibility	$20-30M	3-5 years	Could shift governance strategy entirely	High
Alignment tax trajectory	$20-40M	2-3 years	Could change safety adoption by 50-80%	High
Warning shot response time	$5-10M	Historical analysis	Could change intervention timing strategy	Medium

Resource Implications

Prioritize research that resolves high-leverage uncertainties:

Scaling law empirics: Fund large-scale capability forecasting ($50-100M/year)
Deception detection: Accelerate interpretability and evaluation research ($30-50M/year)
Governance feasibility studies: Diplomatic track-2 engagement, scenario planning ($20-30M/year)
Historical case study analysis: Rapid response literature, regulatory speed ($5-10M/year)

Recommended uncertainty-resolution research budget: $100-200M/year (vs. ≈$20-30M current).

Key Cruxes

Crux	Resolution Method	If Resolved Favorably	If Resolved Unfavorably
Scaling laws continue	Empirical extrapolation	Standard timeline applies	Faster or slower than expected
Alignment tax is reducible	Technical research	Safety adoption accelerates	Racing dynamics intensify
Warning shots are informative	Historical analysis	Governance window exists	Must act on priors
International coordination possible	Diplomatic engagement	Global governance viable	Fragmented response

Limitations

This model has several important limitations that users should consider when applying these estimates:

Parameter independence assumption. The model treats many variables as conditionally independent when in reality they may be deeply correlated. For example, "alignment tax" and "interpretability progress" likely share underlying drivers (researcher talent, algorithmic insights) that create correlations not captured in the causal graph. Sensitivity analyses should explore correlated parameter movements.

Expert survey limitations. Much of the underlying data comes from expert surveys, which have known biases: AI researchers may systematically under- or over-estimate risks in their own field; sample selection may exclude important perspectives; and framing effects can significantly shift probability estimates. The AI Impacts survey↗ notes that question phrasing affected timeline estimates by 13-48 years.

Rapidly changing ground truth. Parameter estimates become outdated quickly. Metaculus AGI forecasts shifted from 50 years (2020) to ~5 years (2024)—a factor of 10 change in four years. Users should check original sources for current values rather than relying on point estimates from this page.

Missing variables. The 35 variables selected represent a judgment call about what matters most. Potentially important factors not included: specific geopolitical events, individual actor decisions, Black Swan technological breakthroughs, and cultural/social dynamics that affect AI adoption and regulation.

Quantification precision. Many estimates (e.g., "deception detection rate: 30%") represent rough order-of-magnitude guesses rather than empirically grounded values. The uncertainty ranges may themselves be overconfident about our ability to bound these quantities.

This critical uncertainties framework connects to several other analytical tools in the knowledge base:

Related Model	Relationship
Racing Dynamics Impact	Explores economic/strategic variables in depth
AI Capability Threshold Model	Details capability threshold variables
Lab Incentives Model	Analyzes safety funding and coordination dynamics
AI Risk Warning Signs Model	Expands on warning shot and response lag variables
International Coordination Game	Deep dive on US-China and multilateral dynamics

Sources

References

1AI Governance Statistics: Global Trends and Organizational Risks (2025)allaboutai.com▸

A statistics-focused overview of the global AI governance landscape in 2025, highlighting rapid growth in AI regulation across countries alongside alarming gaps in organizational governance practices. Key findings include a 21.3% rise in legislative AI mentions across 75 countries and that 97% of organizations experiencing AI-related breaches lacked proper access controls.

allaboutai.com

2AI Impacts 2023 surveyarXiv·Katja Grace et al.·2024·Paper▸

The 2023 AI Impacts survey gathered predictions from 2,778 AI researchers on the pace and impacts of AI progress. Researchers forecast at least 50% probability of AI systems achieving several advanced capabilities by 2028, and estimate a 50% chance of AI outperforming humans in all tasks by 2047—13 years earlier than the previous year's survey. While 68% of respondents were net optimistic about superhuman AI outcomes, substantial majorities expressed concern about extinction risks and other advanced AI scenarios. There was broad consensus that AI risk research should be prioritized more, despite disagreement about whether faster or slower AI progress would be preferable.

★★★☆☆

arxiv.org

3Pew Research: Public and AI ExpertsPew Research Center▸

A Pew Research Center study comparing attitudes of U.S. adults and AI experts toward artificial intelligence, covering optimism about AI's future, concerns about job displacement, and views on regulation. The study reveals notable divergences between expert and public perspectives on AI risks and benefits.

★★★★☆

pewresearch.org

4Mechanistic Interpretability for AI Safety — A Reviewleonardbereska.github.io▸

A comprehensive academic review by Bereska and Gavves (University of Amsterdam, 2024) that surveys mechanistic interpretability—the practice of reverse-engineering neural networks into human-understandable algorithms—with explicit focus on its relevance to AI safety. The review covers foundational concepts like features and circuits, methodologies for causal dissection of model behaviors, and assesses both the benefits and risks of mechanistic interpretability for alignment. It also identifies key challenges around scalability, automation, and generalization to domains beyond language.

leonardbereska.github.io

5Can AI Scaling Continue Through 2030? (Epoch AI)Epoch AI▸

Epoch AI analyzes the key constraints and bottlenecks that could limit continued AI scaling through 2030, examining factors such as compute availability, energy infrastructure, data availability, and algorithmic progress. The analysis assesses whether current scaling trends in large language models and other AI systems can realistically be sustained over the next several years.

★★★★☆

epoch.ai

6The AI Safety Crisis Hiding Behind Trillion-Dollar ValuationsMedium▸

The article argues that despite soaring AI company valuations in the trillions of dollars, investment in AI safety research remains critically underfunded relative to capabilities development. It highlights the structural and incentive-driven reasons why safety spending lags behind, and calls attention to the dangerous gap between commercial AI deployment pace and safety assurance.

★★☆☆☆

medium.com

7International AI Safety Report 2025internationalaisafetyreport.org▸

A landmark international scientific assessment co-authored by 96 experts from 30 countries, providing a comprehensive overview of general-purpose AI capabilities, risks, and risk management approaches. It aims to establish shared scientific understanding across nations as a foundation for global AI governance. The report covers topics including capability evaluation, misuse risks, systemic risks, and mitigation strategies.

internationalaisafetyreport.org

8An Overview of the AI Safety Funding SituationLessWrong·Stephen McAleese·2023▸

This post provides a comprehensive analysis of global AI safety funding landscape, mapping major funders such as Open Philanthropy and the Survival and Flourishing Fund. It estimates total annual spending on AI safety research and examines whether the field is primarily constrained by funding availability or by a shortage of qualified talent.

★★★☆☆

lesswrong.com

9Metaculus Question: When Will the First General AI System Be Devised, Tested, and Publicly Announced?Metaculus▸

A Metaculus forecasting question asking community members to predict when the first generally capable AI system will be created and publicly announced. The question aggregates probabilistic forecasts from many forecasters to produce a community estimate on AGI arrival timing.

★★★☆☆

metaculus.com

10McKinsey State of AI 2025McKinsey & Company▸

McKinsey's annual survey-based report tracking enterprise AI adoption, investment trends, and organizational practices across industries. It provides data on how companies are deploying AI, where value is being generated, and emerging risks and governance challenges associated with scaling AI systems.

★★★☆☆

mckinsey.com

11IAPP AI Governance in Practice Reportiapp.org▸

The IAPP AI Governance in Practice Report surveys how organizations are operationalizing AI governance frameworks, covering policy implementation, risk management, and compliance practices. It provides practical insights into how privacy and data professionals are addressing AI oversight challenges across industries. The report serves as a benchmark for understanding the current state of enterprise AI governance adoption.

iapp.org

12Stanford AI Index 2025Stanford HAI▸

The 2025 Stanford HAI AI Index Report provides a comprehensive annual survey of AI development across technical performance, economic investment, global competition, and responsible AI adoption. It synthesizes data from academia, industry, and government to track AI progress and societal impact. The report serves as a key reference for understanding where AI stands today and emerging trends shaping the field.

★★★★☆

hai.stanford.edu

1380,000 Hours AGI Timelines Review80,000 Hours·Benjamin Todd·2025▸

A comprehensive synthesis by 80,000 Hours reviewing expert predictions on AGI timelines from multiple groups including AI lab leaders, researchers, and forecasters. The review finds a notable convergence toward shorter timelines, with many estimates suggesting AGI could arrive before 2030. Different expert communities that previously disagreed are now showing increasingly similar estimates.

★★★☆☆

80000hours.org

14Preparedness FrameworkOpenAI▸

OpenAI's Preparedness Framework outlines a structured approach to evaluating and managing catastrophic risks from frontier AI models, including threats related to CBRN weapons, cyberattacks, and loss of human control. It defines risk severity thresholds and ties model deployment decisions to safety evaluations. The framework represents OpenAI's operational policy for responsible frontier model development.

★★★★☆

openai.com

AI Risk Critical Uncertainties Model