AI Structural Risk Cruxes

Quick Assessment

Key Links

What Are Structural Risk Cruxes?

Structural risks from AI—including power concentration, lock-inRiskAI Value Lock-inComprehensive analysis of AI lock-in scenarios where values, systems, or power structures become permanently entrenched. Documents evidence including Big Tech's 66-70% cloud control, AI surveillanc...Quality: 64/100 of values or institutions, and breakdown of human agency—represent some of the most consequential yet uncertain challenges posed by advanced artificial intelligence. Unlike traditional AI safety risks focused on specific system failures, structural risks concern how AI transforms the fundamental architecture of human civilization. Your position on key uncertainties, or "cruxes," in this domain largely determines whether you view these risks as urgent priorities requiring immediate governance interventions, or as speculative concerns that shouldn't distract from more concrete technical safety work.

These cruxes are particularly important because they operate at different levels of abstraction and timescales. Some concern foundational questions about whether structural risks constitute a meaningful analytical category distinct from accident and misuse risks. Others focus on near-term competitive dynamics between AI developers and nations. Still others examine long-term questions about technological lock-in and human agency that may unfold over decades. The positions you take on these uncertainties collectively determine your overall structural risk worldview and corresponding intervention priorities.

Given the conceptual fuzziness inherent in structural risk analysis, these cruxes are themselves more speculative than those in other AI safety domains. Many lack clear empirical resolution criteria and involve complex interactions between technological capabilities, social dynamics, and institutional responses. Nevertheless, they represent the key decision points that separate different approaches to understanding and addressing AI's systemic implications for human civilization.

Crux Decision Framework

flowchart TD
  START[Structural Risk Assessment] --> FOUND[Foundational Cruxes]
  FOUND --> Q1{Are structural risks<br/>distinct from accident/misuse?}
  Q1 -->|Yes 40-55%| COORD[Competition & Coordination Cruxes]
  Q1 -->|No 15-25%| TECH[Focus on Technical Safety]
  COORD --> Q2{Can coordination<br/>prevent racing?}
  Q2 -->|Yes 35-45%| GOV[Governance Interventions]
  Q2 -->|No 30-45%| DEF[Defensive Measures]
  GOV --> Q3{Is international<br/>coordination feasible?}
  Q3 -->|Yes 15-30%| INTL[International Frameworks]
  Q3 -->|No 25-40%| DOM[Domestic Governance]
  DEF --> POWER[Power & Lock-in Cruxes]
  DOM --> POWER
  INTL --> POWER
  POWER --> Q4{Is lock-in<br/>reversible?}
  Q4 -->|Yes 35-45%| ADAPT[Build Adaptability]
  Q4 -->|No 20-35%| PREVENT[Prevent Lock-in Now]

  style START fill:#e1f5fe
  style TECH fill:#ffecb3
  style GOV fill:#c8e6c9
  style DEF fill:#ffcdd2
  style PREVENT fill:#ffcdd2
  style ADAPT fill:#c8e6c9

This decision tree illustrates how positions on foundational cruxes cascade into different strategic priorities. The percentages represent rough probability ranges for each position based on expert elicitation.

Foundational Cruxes

This foundational crux shapes the entire field's approach to AI safety prioritization. Those who view structural risks as genuinely distinct argue that AI's effects on power concentration, institutional stability, and human agency operate through different causal mechanisms than individual system failures. They point to examples like algorithmic bias in hiring creating systematic inequality, or AI-enabled surveillance transforming state-citizen relationships—phenomena that emerge from the aggregate deployment of AI systems rather than specific malfunctions. This position suggests structural interventions like governance frameworks, coordination mechanisms, and institutional reforms are necessary complements to technical safety work.

Alternatively, researchers who view structural risks as primarily an aggregation of individual risks argue that focusing on preventing accidents and misuse will naturally address structural concerns. They contend that "structural risk" often conflates correlation with causation, attributing to AI what may simply reflect broader technological and social trends. This perspective suggests that the structural framing may obscure more concrete intervention points and dilute resources from proven technical safety approaches.

Evidence on AI Market Concentration

Recent research provides quantitative evidence on AI's power-concentrating effects:

In July 2024, the DOJ, FTC, UK CMA, and European Commission released a joint statement↗🏛️ government★★★★★US CongressCompetition and Antitrust Concerns Related to Generative AIA CRS report providing U.S. Congressional staff with background on antitrust and competition policy issues in generative AI; useful for understanding the regulatory landscape and legislative debates around AI market concentration and compute access.This Congressional Research Service report analyzes how the high computational costs of developing large language models may create barriers to entry that favor well-resourced t...governancepolicycomputecapabilities+2Source ↗ specifying three competition concerns: concentrated control of key inputs (chips, compute, talent), incumbent digital firms extending power into AI markets, and arrangements among key players reducing competition.

Evidence for AI's distinctive power-concentrating effects includes its scalability without proportional resource increases, network effects where data advantages compound, and first-mover advantages in setting industry standards. Current AI development shows extreme concentration among a handful of companies with the computational resources for frontier model training—a pattern that may be more pronounced than previous technologies. The transformative nature of general intelligence could amplify these effects beyond historical precedent.

However, historical analysis reveals that many transformative technologies initially appeared to concentrate power dramatically before competitive forces and regulatory responses distributed benefits more widely. The printing press, telegraph, and internet all raised similar concerns about information control and market concentration. Some economists argue that AI follows familiar patterns of innovation diffusion, where initial concentration gives way to broader adoption as costs decrease and capabilities standardize.

Competition and Coordination Cruxes

Current evidence shows clear competitive pressures driving rapid AI development with limited safety coordination. Major labs regularly announce accelerated timelines and capability breakthroughs in apparent response to competitors. The hundreds of billions invested in AI development, combined with first-mover advantages in key markets, creates strong incentives to prioritize speed over safety measures. Geopolitically, the framing of AI as a national security priority further intensifies racing dynamics between the US and China.

Those who believe racing can be managed point to successful coordination in other high-stakes domains, including nuclear weapons control, climate agreements, and financial regulation. They argue that shared recognition of catastrophic risks can overcome competitive pressures when appropriate mechanisms exist. Recent initiatives like responsible scaling policies (RSPs) and voluntary commitments on frontier AI safety represent early attempts at such coordination. However, skeptics note that these voluntary measures lack enforcement mechanisms and may not hold under severe competitive pressure.

Early evidence on voluntary coordination shows mixed results. Anthropic, OpenAI, and other major labs have adopted responsible scaling policies and participated in safety commitments, demonstrating some willingness to coordinate. However, these commitments remain largely aspirational, with limited transparency about implementation and no binding enforcement mechanisms. The recent acceleration in capability announcements and deployment timelines suggests competitive pressures may be overwhelming voluntary restraint.

Industry observers note that successful voluntary coordination often requires repeated interaction, shared norms, and credible monitoring—conditions that may be difficult to maintain in a rapidly evolving field with high stakes. Financial sector coordination during crises provides some positive precedents, but typically involved regulatory backstops and shared crisis recognition. The challenge for AI coordination is achieving cooperation before crises demonstrate the need for restraint.

The current US-China relationship on AI combines strategic competition with limited cooperation on specific issues. While broader technology export controls and investment restrictions reflect deep mistrust, both countries have participated in international AI governance forums and expressed concern about catastrophic risks. The November 2023 Biden-Xi summit produced modest commitments to AI risk dialogue, though follow-through remains limited.

Historical precedents suggest both possibilities and constraints. Nuclear arms control succeeded despite Cold War tensions, demonstrating that existential risks can motivate cooperation even between adversaries. However, those agreements emerged after decades of crisis and near-misses that demonstrated mutual vulnerability. AI cooperation may require similar crisis recognition, which could come too late to prevent harmful racing dynamics.

US-China AI Governance Timeline

Despite these diplomatic milestones, fundamental tensions persist. The US ties AI exports to political alignment through chip export controls, while China promotes "open cooperation with fewer conditions↗🔗 webopen cooperation with fewer conditionsPublished on China-US Focus, a platform associated with the China-United States Exchange Foundation; relevant for understanding geopolitical dimensions of AI governance competition that affect global safety standard adoption.This article analyzes the divergent AI development strategies of the United States and China, contrasting the US approach of conditional cooperation with China's model of more o...governancepolicycoordinationcapabilities+2Source ↗." Former Google CEO Eric Schmidt has called for explicit US-China collaboration↗🔗 web★★★☆☆TechPolicy.Presscalled for explicit US-China collaborationPublished on Tech Policy Press, this piece is relevant to researchers and policymakers interested in international AI governance and the geopolitical dimensions of AI safety coordination, particularly US-China dynamics.This article examines the prospects and challenges of US-China collaboration on AI governance, arguing that despite intense geopolitical competition, structured bilateral engage...governancecoordinationpolicyai-safety+3Source ↗, stating both nations have "a vested interest to keep the world stable" and ensure "human control of these tools."

Power and Lock-in Cruxes

Current evidence shows significant concentration in frontier AI capabilities among a small number of well-resourced companies, driven by advantages in computing resources, data access, and talent acquisition. The enormous costs of training state-of-the-art models—potentially reaching hundreds of millions or billions of dollars—create substantial barriers to entry. Network effects and data advantages may further compound these inequalities, as successful AI systems generate user data that improves performance.

However, the trajectory toward winner-take-all outcomes remains uncertain. Open-source AI development has produced capable models like Llama and others that approach frontier performance at lower costs. Regulatory intervention could limit concentration through antitrust enforcement or mandatory sharing requirements. Historical precedent suggests that even technologies with strong network effects often settle into competitive oligopolies rather than pure monopolies.

The permanence of potential AI-enabled lock-in depends on several factors that remain highly uncertain. Advanced AI systems could theoretically enable unprecedented surveillance and control capabilities, making coordination for change extremely difficult. If AI development concentrated among a small number of actors, they might gain sufficient leverage to preserve favorable arrangements indefinitely. The speed and scale of AI deployment could create path dependencies that become increasingly difficult to reverse.

However, historical analysis suggests that even seemingly permanent institutional arrangements eventually face challenges from technological change, generational shifts, or external pressures. The Soviet system appeared locked-in for decades before rapid collapse. Economic and technological evolution continues to create new possibilities for social organization. The question may be not whether AI-enabled lock-in would be reversible, but whether it would persist long enough to significantly constrain human development.

Research on Value Lock-in Mechanisms

Recent research has identified specific mechanisms through which AI could enable value lock-in:

The Forethought Foundation's analysis↗🔗 webForethought Foundation's analysisPublished by Forethought Foundation, this resource addresses lock-in risks from AGI — a concern prominent in longtermist AI safety discourse, relevant to researchers thinking about societal-scale impacts and governance structures for transformative AI.This Forethought Foundation research examines the risks of lock-in scenarios associated with the development of AGI, exploring how advanced AI systems could entrench particular ...existential-riskgovernanceai-safetyalignment+3Source ↗ notes that AGI could make it "technologically feasible to perfectly preserve nuanced specifications of a wide variety of values or goals far into the future"—potentially for "millions, and plausibly trillions, of years." The World Economic Forum's 2024 white paper on AI Value Alignment↗🔗 web★★★★☆World Economic ForumWorld Economic Forum's 2024 white paper on AI Value AlignmentA high-level policy-oriented WEF white paper aimed at business and government audiences; useful for understanding mainstream institutional framing of alignment but less technically rigorous than academic AI safety literature.A World Economic Forum white paper examining the challenge of aligning AI systems with human values, exploring technical and governance frameworks for ensuring AI behaves in acc...alignmentgovernanceai-safetypolicy+4Source ↗ explores how to guide AI systems toward shared human values while preserving adaptability.

Concerns about premature values crystallization reflect the observation that AI systems necessarily embed particular values and assumptions in their design and training. If these systems become sufficiently powerful and widespread, they might entrench current moral frameworks before humanity has time to develop greater moral wisdom through experience and reflection. Historical examples of moral progress—such as expanding circles of moral consideration or evolving concepts of justice—suggest that continued value evolution is important for human flourishing.

Critics argue that values crystallization concerns may be overblown, pointing to the continued evolution of values even in stable societies with established institutions. They note that AI systems can be updated and retrained as values evolve, and that competitive pressures may favor systems aligned with evolving social preferences. The challenge lies in distinguishing between values that should be preserved and those that should remain open to evolution.

Human Agency Cruxes

Evidence from aviation automation provides concerning precedents for skill atrophy concerns. Pilots who rely heavily on autopilot systems show measurable deterioration in manual flying skills, contributing to accidents when automation fails and human intervention is required. Similar patterns appear in navigation (GPS dependency), calculation (calculator reliance), and memory (smartphone externalization). The concern is that widespread AI assistance could create systemic vulnerability if humans lose capacity for independent judgment and action.

However, automation also demonstrates that humans can maintain critical skills through deliberate practice and appropriate system design. Airlines mandate manual flying requirements and emergency procedures training. Medical professionals maintain diagnostic skills despite decision support systems. The key question is whether society will proactively identify and preserve essential human capabilities, or allow market pressures to optimize for short-term efficiency at the expense of long-term resilience.

Quantitative Evidence on AI-Induced Skill Atrophy

A 2024 paper titled "The Paradox of Augmentation: A Theoretical Model of AI-Induced Skill Atrophy↗🔗 web★★★☆☆SSRNThe Paradox of Augmentation: A Theoretical Model of AI-Induced Skill AtrophyTheoretical research examining how AI augmentation may paradoxically lead to skill degradation in human workers, relevant to understanding long-term workforce impacts and human-AI interaction dynamics in AI safety.Venkat Ram Reddy Ganuthula (2024)4 citationsSource ↗" directly addresses the concern that skills erode as humans rely on AI augmentation. Research published in New Biotechnology (2025) by Holzinger et al.↗📄 paper★★★★☆PubMed Central (peer-reviewed)Is human oversight to AI systems still possible?Published in a peer-reviewed medical/life sciences venue (PubMed), this paper addresses the broader question of human oversight viability, relevant to AI safety researchers and policymakers concerned with maintaining meaningful human control over increasingly autonomous systems.Andreas Holzinger, Kurt Zatloukal, Heimo Müller (2025)This paper examines the feasibility of meaningful human oversight over increasingly capable AI systems, analyzing the technical, organizational, and institutional challenges tha...ai-safetygovernancealignmenttechnical-safety+4Source ↗ examines challenges of human oversight in complex AI systems, noting that "as AI systems grow increasingly complex, opaque, and autonomous, ensuring responsible use becomes a formidable challenge."

Current human oversight of AI systems often resembles "security theater"—superficial review procedures that provide reassurance without meaningful control. Large language models operate as black boxes even to their creators, making genuine oversight extremely challenging. As systems become more capable and operate faster than human cognition, maintaining meaningful human involvement becomes increasingly difficult.

Research in interpretability and AI evaluation offers some hope for maintaining oversight through better tools and methodologies. Techniques like mechanistic interpretability, constitutional AI, and automated evaluation could potentially scale human oversight capabilities. However, this requires significant investment and may lag behind capability development. The fundamental challenge is that truly advanced AI systems may operate in ways that exceed human comprehension, making oversight qualitatively different from previous technologies.

Systemic Dynamics Cruxes

The current pace of AI development clearly outpaces institutional adaptation. Regulatory frameworks lag years behind technological capabilities, with agencies struggling to understand systems that evolve monthly. Traditional policy-making processes involving extensive consultation, analysis, and legislative approval are poorly suited to rapidly changing technologies. The result is a governance gap where powerful AI systems operate with minimal oversight or accountability.

However, institutions have demonstrated adaptability to other technological disruptions. Financial regulators responded to digital trading, privacy laws evolved to address internet technologies, and safety standards adapted to new transportation methods. The question is whether AI's pace and breadth of impact exceeds institutional adaptation capacity, or whether new governance approaches can bridge the gap. Experiments in adaptive regulation, regulatory sandboxes, and anticipatory governance offer potential models but remain largely untested at scale.

Institutional Adaptation Approaches

Two contrasting models have emerged for AI governance institutions:

Key 2024-2025 developments include:

• May 2024: Council of Europe adopted first international AI treaty↗📄 paper★★★★★Nature (peer-reviewed)international AI treatyThis Nature journal article analyzes international AI governance challenges and regulatory frameworks, directly addressing AI safety concerns through the lens of international law, regulatory coordination, and enforcement mechanisms needed for responsible AI development.M. Gartner (2024)This article examines the challenges of establishing international AI governance frameworks in a rapidly evolving regulatory landscape. The authors argue that while AI's borderl...Source ↗ on human rights and democracy
• 2024: UN established High-Level Advisory Body on AI
• 2024: Seoul Summit produced voluntary Frontier AI Safety Commitments↗🔗 webFrontier AI Safety CommitmentsSection 1.3 of Dan Hendrycks' open-access AI safety textbook; suitable for foundational learning about how competitive dynamics between states and corporations contribute to catastrophic AI risk scenarios.This textbook chapter from the CAIS 'Introduction to AI Safety, Ethics and Society' covers competitive AI race dynamics, including military AI arms races (lethal autonomous weap...ai-safetyexistential-riskcoordinationgovernance+3Source ↗ from 16 major AI companies
• 2024: Federal AI Risk Management Act mandated NIST AI Risk Management Framework for US agencies

Financial markets provide clear examples of how AI speed can create systemic risks. Flash crashes driven by algorithmic trading have caused market disruptions within milliseconds, too fast for human intervention. These events demonstrate how AI systems interacting at superhuman speeds can create cascading failures that exceed traditional risk management capabilities.

As AI systems become more prevalent across critical infrastructure, similar dynamics could emerge in power grids, transportation networks, or communication systems. The concern is not just individual system failures, but emergent behaviors from AI systems interacting faster than human operators can monitor or control. However, the same speed that creates risks also enables rapid response systems and fail-safes that could mitigate dangers more effectively than human-speed systems.

Safety Implications and Trajectory

The structural risks landscape presents both concerning and promising developments. On the concerning side, current trends show accelerating AI capabilities development with limited coordination between major players, increasing concentration of power among a few well-resourced organizations, and institutional adaptation lagging significantly behind technological change. The competitive dynamics between the US and China have intensified rather than leading to cooperation, while voluntary coordination mechanisms remain largely untested under serious pressure.

However, promising developments include growing awareness of structural risks among policymakers and researchers, early experiments in governance frameworks like responsible scaling policies, and increasing investment in AI safety research including interpretability and alignment work. Some international dialogue on AI governance continues despite broader geopolitical tensions, and civil society organizations are mobilizing around AI accountability and democratic governance issues.

Looking ahead 1-2 years, we expect continued rapid capability development with periodic attempts at voluntary coordination among leading labs. Regulatory frameworks will likely emerge in major jurisdictions but may struggle to keep pace with technological advancement. International coordination will probably remain limited to narrow technical cooperation rather than comprehensive governance regimes. The critical question is whether early warning signs of structural risks will motivate more serious coordination efforts or be dismissed as competitive disadvantage.

In the 2-5 year timeframe, the resolution of several key cruxes may become clearer. We will have better evidence on whether voluntary industry coordination can survive competitive pressures, whether human oversight can scale with AI capabilities, and whether institutions can develop adaptive governance mechanisms. The trajectory of US-China relations and broader geopolitical stability will significantly influence the possibility for international cooperation. Most importantly, we may see the first examples of AI systems with capabilities that clearly exceed human oversight capacity, forcing concrete decisions about acceptable risk levels and governance approaches.

Key Uncertainties

Despite extensive analysis, fundamental uncertainties remain about structural risks from AI. We lack clear empirical metrics for measuring power concentration or institutional adaptation speed, making it difficult to distinguish normal technological disruption from qualitatively new structural changes. The interaction effects between technical AI capabilities and social dynamics are poorly understood, with most analysis based on speculation rather than rigorous empirical study.

The timeline for critical decisions remains highly uncertain. Some structural changes may happen gradually over decades, allowing time for institutional adaptation, while others could occur rapidly during periods of capability growth or geopolitical crisis. We also have limited understanding of which interventions would be most effective, with ongoing debates about whether technical solutions, governance frameworks, or democratic accountability measures should take priority.

Perhaps most fundamentally, the very definition and boundaries of structural risks remain contested. This conceptual uncertainty makes it difficult to design targeted interventions or evaluate progress. Resolution of these foundational questions will likely require both theoretical development and empirical evidence from AI deployment at scale—evidence that may come too late to prevent potentially harmful structural changes.

Position Implications

Sources and Further Reading

Academic Research

• Korinek & Vipra (2025): Concentrating Intelligence: Scaling and Market Structure in AI↗🔗 webConcentrating Intelligence: Scaling and Market Structure in Artificial IntelligenceA 2024 INET working paper offering an economics-focused analysis of AI market concentration; relevant to governance discussions about antitrust, compute access, and structural risks from dominant AI providers.Korinek and Vipra analyze the competitive dynamics and market structure of foundation model markets, focusing on LLMs. They identify economies of scale and scope in compute, dat...governancepolicycapabilitiescompute+3Source ↗ - Economic analysis of AI market concentration
• Gans (2024): Market Power in Artificial Intelligence↗🔗 web★★★★☆NBERGans (2024): Market Power in Artificial IntelligenceUseful reference for AI governance discussions around data monopolies and market concentration; relevant to policymakers and researchers studying structural risks from AI industry consolidation, though not directly focused on safety or alignment.Joshua Gans surveys economic literature on competition and market power across three AI market segments: training data, input data, and AI predictions. The paper argues that the...governancepolicycapabilitiesdeployment+2Source ↗ - NBER analysis of competition drivers
• Ganuthula (2024): The Paradox of Augmentation↗🔗 web★★★☆☆SSRNThe Paradox of Augmentation: A Theoretical Model of AI-Induced Skill AtrophyTheoretical research examining how AI augmentation may paradoxically lead to skill degradation in human workers, relevant to understanding long-term workforce impacts and human-AI interaction dynamics in AI safety.Venkat Ram Reddy Ganuthula (2024)4 citationsSource ↗ - Theoretical model of AI-induced skill atrophy
• Holzinger et al. (2025): Is human oversight to AI systems still possible?↗📄 paper★★★★☆PubMed Central (peer-reviewed)Is human oversight to AI systems still possible?Published in a peer-reviewed medical/life sciences venue (PubMed), this paper addresses the broader question of human oversight viability, relevant to AI safety researchers and policymakers concerned with maintaining meaningful human control over increasingly autonomous systems.Andreas Holzinger, Kurt Zatloukal, Heimo Müller (2025)This paper examines the feasibility of meaningful human oversight over increasingly capable AI systems, analyzing the technical, organizational, and institutional challenges tha...ai-safetygovernancealignmenttechnical-safety+4Source ↗ - New Biotechnology analysis of oversight challenges
• AI Governance in a Complex Regulatory Landscape↗📄 paper★★★★★Nature (peer-reviewed)international AI treatyThis Nature journal article analyzes international AI governance challenges and regulatory frameworks, directly addressing AI safety concerns through the lens of international law, regulatory coordination, and enforcement mechanisms needed for responsible AI development.M. Gartner (2024)This article examines the challenges of establishing international AI governance frameworks in a rapidly evolving regulatory landscape. The authors argue that while AI's borderl...Source ↗ - Humanities and Social Sciences Communications global perspective

Policy Reports

• Congressional Research Service: Competition and Antitrust Concerns Related to Generative AI↗🏛️ government★★★★★US CongressCompetition and Antitrust Concerns Related to Generative AIA CRS report providing U.S. Congressional staff with background on antitrust and competition policy issues in generative AI; useful for understanding the regulatory landscape and legislative debates around AI market concentration and compute access.This Congressional Research Service report analyzes how the high computational costs of developing large language models may create barriers to entry that favor well-resourced t...governancepolicycomputecapabilities+2Source ↗ - 2024 analysis of US competition issues
• AI Now Institute: Artificial Power↗🔗 web★★★★☆AI Now InstituteAI Now Institute: Artificial PowerPublished by the AI Now Institute, a prominent research organization focused on social implications of AI; this report represents a critical, power-focused perspective on AI governance often underrepresented in technical AI safety discussions.The AI Now Institute's 'Artificial Power' report examines how AI development is concentrating economic and political power in a small number of large technology corporations, an...governancepolicyai-safetydeployment+2Source ↗ - Concentration and power in AI
• Open Markets Institute: AI and Market Concentration↗🔗 webOpen Markets Institute: AI and Market ConcentrationPublished by the Open Markets Institute, a competition policy think tank; relevant for understanding structural and antitrust dimensions of AI governance, particularly around compute and platform concentration risks.This expert brief from the Open Markets Institute examines how the rapid development of AI technologies is accelerating market concentration, allowing a small number of dominant...governancepolicycapabilitiescoordination+2Source ↗ - Expert brief on concentration concerns
• Carnegie Endowment: The AI Governance Arms Race↗🔗 web★★★★☆Carnegie EndowmentCarnegie Endowment: The AI Governance Arms RaceA Carnegie Endowment policy analysis critiquing the effectiveness of international AI governance summits, relevant for those tracking multilateral AI safety coordination efforts and the gap between diplomatic activity and substantive regulatory progress.This Carnegie Endowment analysis examines the proliferation of international AI governance summits and frameworks, arguing that despite high-profile diplomatic activity, substan...governancepolicycoordinationai-safety+2Source ↗ - Analysis of governance coordination

International Governance

• Sandia National Labs: US-China AI Collaboration Challenges↗🏛️ governmentSandia National Labs: US-China AI Collaboration ChallengesA 2025 Sandia National Laboratories report relevant to AI governance researchers and policymakers tracking US-China relations on AI safety, particularly nuclear-AI intersections and the geopolitics of international AI governance under the Trump administration.This Sandia National Laboratories report analyzes the state of US-China AI governance collaboration, covering domestic policies, bilateral engagement history, and multilateral p...governancepolicycoordinationai-safety+3Source ↗ - 2025 analysis of cooperation barriers
• TechPolicy.Press: From Competition to Cooperation↗🔗 web★★★☆☆TechPolicy.Presscalled for explicit US-China collaborationPublished on Tech Policy Press, this piece is relevant to researchers and policymakers interested in international AI governance and the geopolitical dimensions of AI safety coordination, particularly US-China dynamics.This article examines the prospects and challenges of US-China collaboration on AI governance, arguing that despite intense geopolitical competition, structured bilateral engage...governancecoordinationpolicyai-safety+3Source ↗ - US-China engagement analysis
• China's Global AI Governance Action Plan↗🏛️ governmentChina's Global AI Governance Action PlanThis is China's official government position on global AI governance, significant for understanding the geopolitical landscape of AI regulation and how major AI powers frame safety, sovereignty, and international cooperation norms.China's official AI governance framework, released at the 2025 World AI Conference, establishes principles for international cooperation treating AI as a global public good. It ...governancepolicyai-safetycoordination+2Source ↗ - Ministry of Foreign Affairs (July 2025)

Value Lock-in and Long-term Risks

• Forethought Foundation: AGI and Lock-in↗🔗 webForethought Foundation's analysisPublished by Forethought Foundation, this resource addresses lock-in risks from AGI — a concern prominent in longtermist AI safety discourse, relevant to researchers thinking about societal-scale impacts and governance structures for transformative AI.This Forethought Foundation research examines the risks of lock-in scenarios associated with the development of AGI, exploring how advanced AI systems could entrench particular ...existential-riskgovernanceai-safetyalignment+3Source ↗ - Analysis of permanent value lock-in
• World Economic Forum: AI Value Alignment↗🔗 web★★★★☆World Economic ForumWorld Economic Forum's 2024 white paper on AI Value AlignmentA high-level policy-oriented WEF white paper aimed at business and government audiences; useful for understanding mainstream institutional framing of alignment but less technically rigorous than academic AI safety literature.A World Economic Forum white paper examining the challenge of aligning AI systems with human values, exploring technical and governance frameworks for ensuring AI behaves in acc...alignmentgovernanceai-safetypolicy+4Source ↗ - 2024 white paper on alignment with human values
• The Precipice (Ord, 2020)↗🔗 webOrd (2020): The PrecipiceFoundational longtermist text by Oxford philosopher Toby Ord; frequently cited in AI safety and EA communities as a comprehensive introduction to existential risk, including AI misalignment as a major threat category.Toby Ord's book argues that humanity faces unprecedented existential risks from nuclear weapons, engineered pandemics, and unaligned AI, and that reducing these risks is among t...existential-riskai-safetylongtermismgovernance+3Source ↗ - Framework for existential risk including lock-in
• What We Owe the Future (MacAskill, 2022)↗🔗 webMacAskill (2022): What We Owe the FutureA widely read popular book that brought longtermism and existential risk—including AI risk—into mainstream discourse; foundational for understanding the moral framework underlying much of the EA and AI safety communities' priorities.William MacAskill's book argues for 'longtermism'—the view that positively influencing the long-run future is among the most important moral priorities of our time. It makes the...existential-riskai-safetygovernancecoordination+2Source ↗ - Longtermist perspective on value evolution

Racing Dynamics

• AI Safety Textbook: AI Race↗🔗 webFrontier AI Safety CommitmentsSection 1.3 of Dan Hendrycks' open-access AI safety textbook; suitable for foundational learning about how competitive dynamics between states and corporations contribute to catastrophic AI risk scenarios.This textbook chapter from the CAIS 'Introduction to AI Safety, Ethics and Society' covers competitive AI race dynamics, including military AI arms races (lethal autonomous weap...ai-safetyexistential-riskcoordinationgovernance+3Source ↗ - Comprehensive analysis of competitive dynamics
• TNSR: Debunking the AI Arms Race Theory↗🔗 webTNSR: Debunking the AI Arms Race TheoryPublished in Texas National Security Review (2021) by Paul Scharre, a leading expert on autonomous weapons; relevant to AI governance debates about framing military AI competition and managing associated safety and escalation risks.Paul Scharre argues that the 'AI arms race' framing is inaccurate because AI is a general-purpose technology rather than a weapon, making it fundamentally different from traditi...governancepolicyai-safetydeployment+3Source ↗ - Skeptical perspective on arms race framing
• Bostrom: Racing to the Precipice↗🔗 webBostrom: Racing to the PrecipiceA foundational paper by Nick Bostrom formalizing the 'AI race' concern, frequently cited in discussions about why competitive dynamics between labs or nations may structurally undermine safety efforts and require governance solutions.Bostrom models competitive AI development as a race dynamic where safety-capability tradeoffs can lead rational actors to sacrifice safety for competitive advantage, potentially...ai-safetyexistential-riskcoordinationgovernance+2Source ↗ - Original model of AI development races

Institutional Adaptation

• World Economic Forum: Governance in the Age of Generative AI↗🔗 web★★★★☆World Economic ForumWorld Economic Forum: Governance in the Age of Generative AIA WEF report aimed at global policymakers and business leaders; represents mainstream institutional thinking on AI governance rather than technical AI safety research, useful for understanding the policy landscape surrounding generative AI.This World Economic Forum publication examines frameworks and policy recommendations for governing generative AI technologies at a global level. It addresses the challenges of b...governancepolicycoordinationdeployment+2Source ↗ - 2024 governance framework
• Stanford FSI: Regulating Under Uncertainty↗🔗 webStanford FSI: Regulating Under UncertaintyPublished by Stanford's Cyber Policy Center, this is a policy-oriented analysis aimed at lawmakers and regulators grappling with how to govern generative AI responsibly amid significant technical and social uncertainty.This Stanford FSI report examines governance frameworks for generative AI given deep uncertainty about its risks and capabilities. It analyzes various regulatory approaches and ...governancepolicyai-safetydeployment+2Source ↗ - Governance options analysis
• WEF: GenAI is rapidly evolving↗🔗 web★★★★☆World Economic ForumWorld Economic ForumPublished by the World Economic Forum in October 2024, this piece summarizes a broader AI Governance Alliance report aimed at equipping policymakers with practical tools for GenAI regulation; useful as a policy-oriented governance reference.This World Economic Forum article, tied to a report from the AI Governance Alliance, outlines the challenges governments face in regulating generative AI and presents a practica...governancepolicyai-safetydeployment+2Source ↗ - How governments can keep pace