ForecastBench
activeDynamic, contamination-free benchmark for evaluating LLM forecasting capabilities, published at ICLR 2025.
Organizations
6| Forecasting Research Institute (FRI) | Research institute advancing forecasting methodology through large-scale tournaments and rigorous experiments, led by Philip Tetlock. |
| Bridgewater AIA Labs | Bridgewater AIA Labs launched a $2B AI-driven macro fund in July 2024 that returned 11.9% in 2025, using proprietary ML models plus LLMs from OpenAI/Anthropic/Perplexity with multi-layer guardrails that reduced error rates from 8% to 1.6%. The division has minimal AI safety relevance, focusing on financial applications rather than safety research. |
| Coefficient Giving | Coefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, with 68% going to evaluations/benchmarking, and launched a $40M Technical AI Safety RFP in 2025 covering 8 research areas. |
| Metaculus | Reputation-based prediction aggregation platform that has become the primary source for AI timeline forecasts, with over 1 million predictions across 15,000+ questions. Created by Anthony Aguirre (FLI/FLF President). |
| Manifold (Prediction Market) | Manifold is a play-money prediction market with millions of predictions and ~2,000 peak daily users, showing AGI by 2030 at ~60% vs Metaculus ~45%. Platform scored Brier 0.0342 on 2024 election (vs Polymarket's 0.0296), demonstrating play-money markets can approach real-money accuracy but with systematic liquidity limitations. |
| Polymarket | This is a comprehensive overview of Polymarket as a prediction market platform, covering its history, mechanics, and accuracy, but has minimal relevance to AI safety beyond brief mentions in the EA/forecasting section. While well-documented, it primarily serves as general reference material about a prediction market platform rather than AI safety analysis. |
People
1| Philip Tetlock | Psychologist and forecasting researcher who pioneered the science of superforecasting through the Good Judgment Project, demonstrating that systematic forecasting methods can outperform expert predictions and intelligence analysts. |
Related Projects
5| XPT (Existential Risk Persuasion Tournament) | Four-month structured forecasting tournament bringing together superforecasters and domain experts through adversarial collaboration. |
| AI Forecasting Benchmark Tournament | Quarterly competition run by Metaculus comparing human Pro Forecasters against AI forecasting bots. |
| Metaforecast | Forecast aggregation platform combining predictions from 10+ sources into a unified search interface. |
| Squiggle | Domain-specific programming language for probabilistic estimation with native distribution types and Monte Carlo sampling. |
| SquiggleAI | LLM-powered tool for generating probabilistic models in Squiggle from natural language descriptions. |
Related Wiki Pages
Top Related Pages
Project
XPT (Existential Risk Persuasion Tournament)
A four-month structured forecasting tournament (June-October 2022) that brought together 169 participants—89 superforecasters and 80 domain experts...
Organization
Forecasting Research Institute (FRI)
The Forecasting Research Institute (FRI) advances forecasting methodology through large-scale tournaments and rigorous experiments.
Project
AI Forecasting Benchmark Tournament
A quarterly competition run by Metaculus comparing human Pro Forecasters against AI forecasting bots.
Person
Philip Tetlock
Psychologist and forecasting researcher who pioneered the science of superforecasting through the Good Judgment Project, demonstrating that systema...
Concept
Epistemic & Forecasting Organizations (Overview)
Organizations advancing forecasting methodology, prediction aggregation, and epistemic infrastructure to improve decision-making on AI safety and e...
Key Facts
- Website
- https://www.forecastbench.org
- Founded Date
- Sep 2024
- Founded By
- Forecasting Research Institute (FRI)