Metaforecast

Quick Assessment

Project Details

Overview

Metaforecast aggregates forecasts from multiple prediction platforms into a single searchable interface, solving the platform fragmentation problem that reduces forecasting's utility as a public good. While individual platforms like MetaculusOrganizationMetaculusMetaculus is a reputation-based forecasting platform with 1M+ predictions showing AGI probability at 25% by 2027 and 50% by 2031 (down from 50 years away in 2020). Analysis finds good short-term ca...Quality: 50/100, ManifoldOrganizationManifold (Prediction Market)Manifold is a play-money prediction market with millions of predictions and ~2,000 peak daily users, showing AGI by 2030 at ~60% vs Metaculus ~45%. Platform scored Brier 0.0342 on 2024 election (vs...Quality: 43/100, and PolymarketOrganizationPolymarketThis is a comprehensive overview of Polymarket as a prediction market platform, covering its history, mechanics, and accuracy, but has minimal relevance to AI safety beyond brief mentions in the EA...Quality: 33/100 each host valuable forecasts, researchers and decision-makers must check multiple sites to find relevant predictions. Metaforecast eliminates this friction by combining data from 10+ platforms with unified search.

The project was initiated by Nuño Sempere, with help from Ozzie Gooen, at QURI. Data is fetched daily via automated scraping, showing immediate forecasts without historical time-series data. This design is optimized for quick lookups ("What's the current probability of X?") rather than trend analysis.

Design Philosophy

Metaforecast treats forecasting as a public good that becomes more valuable with aggregation. The platform is deliberately simple: search for a question, see current probabilities from multiple sources, compare platforms side-by-side. No user accounts, no social features, no complexity—just forecasts.

Platforms Aggregated

Metaforecast currently indexes approximately 2,100 active forecasting questions across these platforms:

Additionally, Metaforecast provides access to 17,000+ public models from Guesstimate, Ozzie Gooen's earlier Monte Carlo spreadsheet tool.

Platform Characteristics

Technical Architecture

Data Pipeline

Data Freshness

Metaforecast fetches data daily, ensuring forecasts are current. However, the system is optimized for point-in-time forecasts rather than historical trend analysis:

• Shows: Current probability estimates from each platform
• Does not show: Historical time-series or forecast evolution over time

This design choice reduces storage requirements and simplifies the interface at the cost of trend analysis capabilities.

Features

Unified Search

Search across all platforms simultaneously:

Query: "AGI by 2030"
Results: Shows all matching questions from Metaculus, Manifold, Polymarket, etc.

Platform Comparison

See how different sources estimate the same event:

This comparison helps identify consensus vs. divergence in forecasting communities.

GraphQL API

Programmatic access for integrations:

query {
  questions(limit: 10, query: "AI capabilities") {
    title
    platform
    probability
    url
  }
}

The API enables:

• Custom analysis and visualization
• Integration with research workflows
• Automated monitoring of specific questions

Open Source

The GitHub repository is part of QURI's Squiggle monorepo, allowing community contributions and transparency in the scraping methodology.

Integrations

Metaforecast has been integrated with several external services:

These integrations make forecast data accessible in contexts where users already work.

Use Cases

Research

Researchers use Metaforecast to:

• Find consensus forecasts on emerging technologies
• Compare expert vs. market-based predictions
• Track how different communities view the same question
• Validate assumptions in models with community forecasts

Decision-Making

Organizations reference Metaforecast for:

• Strategic planning timeline assumptions
• Risk assessment probability inputs
• Validation of internal forecasts against external views

Forecasting Meta-Analysis

Researchers studying forecasting accuracy use Metaforecast to:

• Compare platform performance
• Identify systematic biases
• Analyze calibration across question types

Notable Use: 2024 US Election

During the 2024 US presidential election, Polymarket (one of Metaforecast's sources) demonstrated the value of prediction market aggregation. Polymarket strongly favored Trump's victory even as traditional polls showed a closer race. The market's prediction proved correct, with reports of approximately $10 million in bets from individual traders helping shape the odds.

Metaforecast allowed researchers to compare Polymarket's market-based odds against Metaculus's reputation-based aggregation and other sources, providing multiple perspectives on the same event.

Comparison with Individual Platforms

Tradeoffs

Metaforecast gains:

• Convenience: Single search across all platforms
• Comparison: See multiple sources side-by-side
• Completeness: Less likely to miss relevant forecasts

Individual platforms gain:

• Historical data: Track forecast evolution
• Community context: Discussion, reasoning transparency
• Participation: Ability to contribute forecasts

Strengths and Limitations

Strengths

Limitations

Relationship to QURI Ecosystem

Metaforecast complements QURI's other tools:

Funding

As a QURI project, Metaforecast is funded through QURI's grants:

Metaforecast represents a small portion of QURI's overall budget but provides a valuable public good for the forecasting community.

Future Directions

Potential enhancements based on community feedback:

External Links

• Metaforecast
• GitHub Repository
• EA Forum: Introducing Metaforecast
• Nuño Sempere EA Forum Profile
• QURI Website