Davidson & Houlden 2025

Will AI R&D Automation Cause a Software Intelligence Explosion? Will AI R&D Automation Cause a Software Intelligence Explosion? Daniel Eth Tom Davidson Authors Citations Cite Citations PDF Contact 26th March 2025 Will AI R&D Automation Cause a Software Intelligence Explosion? Summary Key Points Introduction Where AI progress comes from Improvements in AI software are already driving fast AI progress AI progress will likely speed up as we approach ASARA What happens when we reach ASARA? A toy model to demonstrate the dynamics of a software intelligence explosion Being more mathematically concrete: returns to software R&D In the real world, are returns to software R&D greater or less than one? You might need fast growing computing power to discover better algorithms Progress might become bottlenecked by the time required to train new AI systems Bringing it all together What can we do if an SIE is possible? References Appendix: Justification for our formulation of r Empirical evidence suggests that, if AI automates AI research, feedback loops could overcome diminishing returns, significantly accelerating AI progress Summary AI companies are increasingly using AI systems to accelerate AI research and development. These systems assist with tasks like writing code, analyzing research papers, and generating training data. While current systems struggle with longer and less well-defined tasks, future systems may be able to independently handle the entire AI development cycle – from formulating research questions and designing experiments, to implementing, testing, and refining new AI systems. Some analysts have argued that such systems, which we call AI Systems for AI R&D Automation (ASARA), would represent a critical threshold in AI development. The hypothesis is that ASARA would trigger a runaway feedback loop: ASARA would quickly develop more advanced AI, which would itself develop even more advanced AI, resulting in extremely fast AI progress – an “intelligence explosion.” Skeptics of an intelligence explosion often focus on hardware limitations – would AI systems be able to build better computer chips fast enough to drive such rapid progress? However, there’s another possibility: AI systems could become dramatically more capable just by finding software improvements that significantly boost performance on existing hardware. This could happen through improvements in neural network architectures, AI training methods, data, scaffolding around AI systems, and so on. We call this scenario a software intelligence explosion (SIE). This type of advancement could be especially rapid, since it wouldn’t be limited by physical manufacturing constraints. Such a rapid advancement could outpace society’s capacity to prepare and adapt. In this report, we examine whether ASARA would lead to an SIE. First, we argue that shortly after ASARA is developed, it will be possible to run orders of magnitude more automated AI researchers than the current number of leading hum

... (truncated, 549 KB total)