Should We Pause AI Development?

In March 2023, the Future of Life Institute published an open letter calling for a 6-month pause on training AI systems more powerful than GPT-4. The letter garnered over 33,000 signatures, including Turing Award winners Yoshua BengioPersonYoshua BengioComprehensive biographical overview of Yoshua Bengio's transition from deep learning pioneer (Turing Award 2018) to AI safety advocate, documenting his 2020 pivot at Mila toward safety research, co...Quality: 39/100 and prominent figures like Elon MuskPersonElon Musk (AI Industry)Comprehensive profile of Elon Musk's role in AI, documenting his early safety warnings (2014-2017), OpenAI founding and contentious departure, xAI launch, and extensive track record of predictions....Quality: 38/100 and Steve Wozniak. It ignited fierce debate: Is pausing AI development necessary for safety, or counterproductive and infeasible?

Quick Assessment

Key Links

The Debate Landscape

The Proposal

Pause advocates call for:

• Moratorium on training runs beyond current frontier (GPT-4 level)
• Time to develop safety standards and evaluation frameworks
• International coordination on AI governance
• Only resume when safety can be ensured

Duration proposals vary:

• 6 months (FLI letter, March 2023)
• Indefinite until safety solved (Eliezer Yudkowsky in TIME, April 2023)
• "Slow down" rather than full pause (moderates like Yoshua Bengio)

The Spectrum of Positions

Key Cruxes

Alternative Proposals

Many propose middle grounds between full pause and unconstrained racing:

Comparison of Alternatives

Detailed Alternatives

Responsible Scaling Policies (RSPs)

• Continue development but with if-then commitments
• If dangerous capabilities detected, implement safeguards or pause
• Anthropic's approach uses AI Safety Levels (ASL-1 through ASL-4+)
• As of May 2025, Anthropic activated ASL-3 for Claude Opus 4 due to CBRN concerns

Compute Governance

• Limit training compute through regulation or voluntary agreement
• US export controls restrict advanced AI chips to China and ~150 other countries
• The EU AI Act defines "high-risk" based on compute thresholds (10^25 FLOP)
• Easier to verify than complete pause—chip production is concentrated in few fabs

Safety Tax

• Require safety work proportional to capabilities
• E.g., spend 20% of compute on safety research
• Maintains progress while prioritizing safety
• No mandatory implementation; relies on voluntary commitment

Staged Deployment

• Develop models but delay deployment for safety testing
• Allows research while preventing premature release

International Registry

• Register large training runs with international body
• Creates visibility without stopping work
• Foundation for future coordination
• Seoul AI Summit (2024) established voluntary commitments for 16 AI companies

Threshold-Based Pause

• Continue until specific capability thresholds (e.g., autonomous replication)
• Then pause until safeguards developed
• Clear criteria, only activates when needed

The Coordination Problem

Why is coordination so hard? Analysis of AI governance challenges suggests coordination failure is the default outcome absent strong institutional mechanisms.

Key Actors and Their Stakes

Verification challenges:

• Training runs are secret—only ~10-20 organizations can train frontier models
• Compute usage is hard to monitor without chip-level tracking
• Open source development involves 100,000+ contributors globally
• PauseAI protests in 13 countries (May 2024) had minimal policy impact

Incentive misalignment:

• First to AGI gains enormous advantage—estimated $1-10T+ value capture
• Defecting from pause very tempting—6-12 month lead could be decisive
• Short-term vs long-term tradeoffs favor short-term action
• National security concerns: US-China AI competition frames pause as "unilateral disarmament"

Precedents suggest pessimism:

But some hope:

• All parties may share existential risk concern—70% of AI researchers want more safety prioritization
• Industry may support regulation to avoid liability and level playing field
• Compute is traceable—TSMC and Samsung produce 90%+ of advanced chips; ASML is sole EUV lithography supplier
• China has expressed interest in international coordination: "only with joint efforts of the international community can we ensure AI technology's safe and reliable development"

What Would Need to Be True for a Pause to Work?

For a pause to be both feasible and beneficial:

Current reality: Few of these conditions are met. As FLI noted on the letter's one-year anniversary, AI companies have instead directed "vast investments in infrastructure to train ever-more giant AI systems."

2024-2025 Developments

The pause debate has evolved significantly since the 2023 letter:

Global AI Governance Initiatives

PauseAI Movement

PauseAI, founded in May 2023 by Dutch software entrepreneur Joep Meindertsma, has organized protests across 13+ countries. Their goals include:

• Temporary pause on training the most powerful general AI systems
• International AI safety agency similar to IAEA
• Democratic control over AI development

Despite ongoing activism, no country has implemented binding pause legislation.

Historical Parallels

Comparison of Technology Governance Precedents

Asilomar Conference on Recombinant DNA (1975):

• Scientists voluntarily paused research on genetic engineering for approximately one year
• ~140 biologists, lawyers, and physicians developed safety guidelines at Pacific Grove, California
• Moratorium was "universally observed" in academic and industrial research centers
• Led to NIH Recombinant DNA Advisory Committee and safety protocols still in use today
• Key difference: Scientists controlled the technology; AI development involves thousands of companies and millions of developers

Nuclear Test Ban Treaties:

• Partial Test Ban Treaty (1963): banned atmospheric testing—verified by detection networks
• Comprehensive Test Ban Treaty (1996): signed by 187 states but not ratified by US, China, or others
• Verification via seismology is feasible; 9 states now possess nuclear weapons
• Key difference: Decades-long timeline allowed governance to develop; AI timelines may be 5-15 years

Ozone Layer (Montreal Protocol):

• Successfully phased out CFCs globally—ozone hole now recovering
• Required finding chemical substitutes (HFCs) and industry buy-in
• Key difference: Clear, measurable environmental indicator; AI risks are speculative and contested

Moratorium on Human Germline Editing:

• Mostly holding after He Jiankui's 2018 violation (3-year prison sentence in China)
• Low economic stakes compared to AI; clear ethical consensus across cultures
• Key difference: AI development has estimated $1-10T+ in value at stake

The Case for "Slowdown" Rather Than "Pause"

Many find middle ground more palatable. Yoshua Bengio, Turing Award winner and lead author of the International AI Safety Report, has advocated for "red lines" that AI systems should never cross rather than a blanket pause:

• Autonomous replication or improvement
• Dominant self-preservation and power seeking
• Assisting in weapon development
• Cyberattacks and deception

Slowdown means:

• Deliberate rather than maximize speed
• Investment in safety alongside capabilities
• Coordination with other labs
• Voluntary agreements where possible

More achievable because:

• Doesn't require stopping completely
• Maintains progress on benefits
• Reduces but doesn't eliminate competition
• Easier political sell

Examples of slowdown mechanisms:

• Labs coordinating on release timing (e.g., OpenAI, Anthropic, Google pre-release safety testing)
• Responsible Scaling Policies with conditional pauses
• Seoul AI Summit commitments from 16 major companies
• EU AI Act compliance requirements (Aug 2025)

Expert Perspectives

Summary of Key Positions

The Disagreement Structure

Most disagreement reduces to different assessments of:

Sources & Further Reading

• Pause Giant AI Experiments: An Open Letter - Future of Life Institute (2023)
• Pausing AI Developments Isn't Enough. We Need to Shut it All Down - Eliezer Yudkowsky, TIME Magazine (2023)
• International AI Safety Report - Yoshua Bengio et al. (2025)
• Anthropic Responsible Scaling Policy - Anthropic (2024)
• 2023 Expert Survey on Progress in AI - AI Impacts (2023)
• Asilomar Conference on Recombinant DNA - Historical precedent (1975)
• Seoul Declaration for Safe, Innovative and Inclusive AI - AI Seoul Summit (2024)
• EU AI Act - European Commission (2024)
• PauseAI - Grassroots movement for AI development pause