Platform calibration | Manifold

[Skip to main content](https://manifold.markets/calibration#main-content)

# Calibration

Explore how Manifold's predictions compare to real-world outcomes. Our track record demonstrates the power of collective forecasting.

📊 Why are markets better than polls or experts?

One paper about [predicting scientific paper replication](https://www.pnas.org/doi/10.1073/pnas.1516179112) compared these forecasting methods. It found that prediction markets outperformed surveys and

> ...could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.”

Either prediction markets are more accurate than experts, or experts should be able to make a lot of money on them, and in doing so correct the markets.

🤑 Are markets resistant to manipulation and hype?

As the market prices moves further from the true probability, the odd's pricing becomes better for traders to correct it in the right direction. Naturally, this increases the incentive to bet accurately as there is more money to be made once the market resolves.

Robin Hanson explores this further in his paper, [A Manipulator Can Aid Prediction Market Accuracy](https://mason.gmu.edu/~rhanson/biashelp.pdf). In it he examines how both historical and lab data fail to find substantial effects of manipulation on average price accuracy. Furthermore, he finds in his model that adding a manipulator may even increase accuracy as it increases noise trading which tends to have a positive effect in low liquidity markets.

See also, [Scott Alexander's failed manipulation attempt](https://www.astralcodexten.com/i/85781340/scandal-markets) on Manifold.

🌱 Do markets with few traders and low liquidity work?

Yes! And very reliably! The paper [Prediction Markets: Practical Experiments in Small Markets and Behaviours Observed](https://core.ac.uk/download/pdf/235244384.pdf), concluded,

> “16 or more traders should be sufficient to obtain quality predictions. Smaller markets may be just as useful, though they may exhibit biases of under confidence toward market favourites.”

Our own [data](https://manifold.markets/vluzko/after-how-many-unique-traders-will#EiWKtYBZaWvQbj27W6tT) has shown that somewhere between 10 - 20 traders our calibration no longer improves with more traders. We still need to conduct analysis on the impact liquidity has on accuracy.

## Overall Calibration

Predicted vs actual outcomes across all markets

0.17422Brier

This chart shows whether events happened as often as we predicted. The closer the blue dots are to the diagonal line, the better our calibration. A dot at 70% on the x-axis should appear at 70% on the y-axis if exactly 70% of those markets resolved yes.

Resolved Yes

Market Probability

Methodology

1. 1Every hour we sample 2% of all past trades on resolved binary questions with 15 or more traders. Current sample size:93ktrades.
2. 2For each sam

... (truncated, 6 KB total)