Back
Why Most Published Research Findings Are False
webAuthor
Sebastian Lobentanzer
Highly relevant to AI safety researchers evaluating empirical claims about model capabilities or alignment techniques, as the statistical biases described apply directly to ML benchmarking and safety evaluation literature.
Metadata
Importance: 72/100journal articleprimary source
Summary
John Ioannidis's landmark 2005 paper demonstrates mathematically that the majority of published research findings are likely false positives, due to low statistical power, publication bias, and researcher degrees of freedom. Using probability modeling, it shows that under common research conditions the post-study probability of a true finding is frequently below 50%. This work catalyzed the modern replication crisis movement across scientific disciplines.
Key Points
- •Uses Bayesian probability modeling to show that most claimed research findings have a higher chance of being false than true under typical study conditions.
- •Key factors inflating false positives include small sample sizes, weak prior probabilities, publication bias toward positive results, and p-hacking.
- •The probability a research finding is true depends heavily on the pre-study odds, statistical power, and the number of relationships tested simultaneously.
- •Fields with greater flexibility in study design, definitions, and outcomes are especially prone to producing false findings.
- •Foundational paper for understanding why AI/ML benchmark results, safety evaluations, and capability claims may not replicate reliably.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Scientific Knowledge Corruption | Risk | 91.0 |
Cached Content Preview
HTTP 200Fetched Mar 15, 202640 KB
Why Most Published Research Findings Are False | PLOS Medicine
Article
Authors
Metrics
Comments
Media Coverage
Reader Comments
Figures
Correction
25 Aug 2022:
Ioannidis JPA
(2022)
Correction: Why Most Published Research Findings Are False.
PLOS Medicine 19(8): e1004085.
https://doi.org/10.1371/journal.pmed.1004085
View correction
Figures
Abstract
Summary
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Citation: Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8):
e124.
https://doi.org/10.1371/journal.pmed.0020124
Published: August 30, 2005
Copyright: © 2005 John P. A. Ioannidis. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Competing interests: The author has declared that no competing interests exist.
Abbreviation:
PPV,
positive predictive value
Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. Refutation and controversy is seen across the range of research designs, from clinical trials and traditional epidemiological studies [ 1–3 ] to the most modern molecular research [ 4 , 5 ]. There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims [ 6–8 ]. However, this should not be surprising. It can be proven that most claimed research findings are false. Here I will examine the key factors that influence this problem a
... (truncated, 40 KB total)Resource ID:
1734a20e751ebd1b | Stable ID: MDNmZDRmY2