Warning Signs Model

Overview

The challenge of AI risk management is fundamentally one of timing: acting too late means risks have already materialized into harms, while acting too early wastes resources and undermines credibility. This model addresses this challenge by cataloging warning signs across different risk categories, distinguishing leading from lagging indicators, and proposing specific tripwires that should trigger predetermined responses. The central question is: What observable signals should prompt us to shift from monitoring to action, and at what thresholds?

Analysis of 32 critical warning signs reveals that most high-priority indicators are 18-48 months from threshold crossing, with detection probabilities ranging from 45-90% under current monitoring infrastructure. However, systematic tracking exists for fewer than 30% of identified warning signs, and pre-committed response protocols exist for fewer than 15%. This gap between conceptual frameworks and operational capacity represents a critical governance vulnerability.

The key insight is that effective early warning systems must balance four competing demands. Early detection requires sensitivity to weak signals, but high sensitivity generates false positives that erode trust and waste resources. Actionable thresholds need specificity to trigger responses but flexibility to accommodate uncertainty. The optimal monitoring system emphasizes leading indicators that predict future risk while using lagging indicators for validation, creating a multi-layered detection architecture that trades off between anticipation and confirmation.

Risk Assessment Table

Conceptual Framework

The warning signs framework organizes indicators along two primary dimensions: temporal position (leading vs. lagging) and signal category (capability, behavioral, incident, research, social). Understanding this structure enables more effective monitoring by clarifying what each indicator type can and cannot tell us about risk trajectories.

flowchart TD
  subgraph Leading["Leading Indicators (Predictive)"]
      CAP[Capability Signals - Benchmark improvements]
      BEH[Behavioral Signals - System behaviors in eval]
      RES[Research Signals - Publications, breakthroughs]
  end

  subgraph Lagging["Lagging Indicators (Confirmatory)"]
      INC[Incident Signals - Real-world events]
      SOC[Social Signals - Institutional responses]
  end

  CAP --> |"Capability enables"| BEH
  BEH --> |"Behavior causes"| INC
  RES --> |"Research drives"| CAP
  INC --> |"Incidents trigger"| SOC
  SOC --> |"Policy affects"| RES

  CAP --> TRP{Tripwire Threshold}
  BEH --> TRP
  INC --> TRP
  TRP --> |"Crossed"| ACT[Predetermined Response]
  TRP --> |"Approaching"| MON[Heightened Monitoring]

Leading indicators predict future risk before it materializes and provide the greatest opportunity for proactive response. Capability improvements on relevant benchmarks signal expanding risk surface before deployment or misuse. Research publications and internal lab evaluations offer windows into near-term trajectories. Policy changes at AI companies can signal anticipated capabilities or perceived risks.

Lagging indicators confirm risk after it begins manifesting and provide validation for leading indicator interpretation. Documented incidents demonstrate theoretical risks becoming practical realities. Economic changes reveal actual impact on labor markets. Policy failures show where existing safeguards proved inadequate. The optimal monitoring strategy combines both types for anticipation and calibration.

Signal Category Framework

The signal categories represent different loci of observation in the AI risk chain. Capability signals are closest to the source and offer the earliest warning, but require the most interpretation. As signals move through behavioral manifestation, real-world incidents, and ultimately social impacts, they become easier to interpret but offer less time for response.

Priority Warning Signs Analysis

Tier 1: Critical Monitoring Gaps

These Tier 1 warning signs combine extreme severity with poor current monitoring, creating dangerous blind spots in risk detection capability.

Tier 2: Moderate Monitoring Gaps

Warning Signs by Risk Category

Control/Alignment Risks

Situational Awareness Indicators

Red lines (immediate action required):

• Clear evidence of training-awareness-based behavior modification
• AI takes actions specifically to evade evaluation
• AI accurately models and exploits oversight gaps

Yellow lines (heightened monitoring):

• Increasing sophistication of self-referential reasoning in chain-of-thought
• Performance gaps between private and public evaluations
• AI questions or challenges evaluation protocols

Deception/Scheming Detection

Red lines:

• AI caught systematically deceiving evaluators about capabilities or goals
• AI takes concrete actions to preserve itself against shutdown
• Evidence of coordination between AI instances for goal preservation

Corrigibility Failure Signs

Misuse Risks

Biological Weapons Capabilities

Critical thresholds:

• AI provides actionable synthesis guidance for CDC Select Agents↗🏛️ governmentFederal Select Agent Program (CDC/USDA)Relevant to AI safety discussions around biosecurity governance, dual-use research risks, and regulatory models that could inform oversight of AI-enabled biological capabilities or gain-of-function research.The Federal Select Agent Program is a joint CDC-USDA regulatory initiative governing the possession, use, and transfer of dangerous biological agents and toxins that could threa...governancepolicybiosecurityexistential-risk+3Source ↗
• Documented synthesis attempt using AI guidance
• Open-source model matches frontier bio capability

Cyber Capabilities

Structural Risks

Economic Displacement Tracking

Epistemic Erosion Indicators

Tripwire Framework

Tripwires are specific, observable conditions that trigger predetermined responses. Unlike general warning signs, tripwires are precisely defined, binary (crossed/not crossed), and linked to specific actions.

Category A: Deployment Pause Tripwires

Category B: Research Escalation Tripwires

Category C: Policy Intervention Tripwires

Monitoring Infrastructure Assessment

Current State Analysis

Required Infrastructure Investment

Total recommended annual investment: $80-200M (currently ≈$15-40M)

Implementation Roadmap

Phase 1: Foundation (Months 1-12)

• Establish AI Incident Database with NIST AI Risk Management Framework↗🏛️ government★★★★★NISTNIST AI Risk Management FrameworkThe NIST AI RMF is a widely referenced U.S. government standard for AI risk governance, frequently cited in policy discussions and used by organizations building internal AI safety and compliance programs; relevant to AI safety researchers tracking institutional governance approaches.The NIST AI RMF is a voluntary, consensus-driven framework released in January 2023 to help organizations identify, assess, and manage risks associated with AI systems while pro...governancepolicyai-safetydeployment+4Source ↗ integration
• Create standardized evaluation protocols through METR↗🔗 web★★★★☆METRMETR: Model Evaluation and Threat ResearchMETR is a leading third-party AI safety evaluation organization whose work on autonomous capability benchmarks and catastrophic risk assessments directly informs AI lab safety policies and government AI governance frameworks.METR is an organization conducting research and evaluations to assess the capabilities and risks of frontier AI systems, focusing on autonomous task completion, AI self-improvem...evaluationred-teamingcapabilitiesai-safety+5Source ↗ and ARC↗🔗 webAlignment Research CenterARC is one of the leading independent technical AI safety research organizations; its evaluations work spun out as METR, and it remains influential in shaping how frontier labs approach pre-deployment safety assessments.The Alignment Research Center (ARC) is a non-profit research organization focused on technical AI alignment and safety research. ARC works on understanding and addressing risks ...ai-safetyalignmenttechnical-safetyinterpretability+5Source ↗
• Implement basic capability tracking via MLPerf↗🔗 webMLCommons - Better AI for EveryoneMLCommons is a key industry body for AI benchmarking and safety measurement; relevant to AI safety researchers interested in standardized evaluation frameworks and governance-by-measurement approaches.MLCommons is an industry-academia consortium of 125+ members focused on developing open, standardized benchmarks and measurement tools for AI performance, safety, and efficiency...evaluationgovernanceai-safetydeployment+3Source ↗ extensions
• Develop tripwire governance frameworks with Partnership on AI↗🔗 web★★★☆☆Partnership on AIPartnership on AI (PAI) – Multi-Stakeholder AI Governance OrganizationPAI is a major multi-stakeholder governance body relevant to AI safety researchers interested in policy coordination, industry norms, and the institutional landscape surrounding responsible AI deployment.Partnership on AI (PAI) is a nonprofit coalition of AI researchers, civil society organizations, academics, and companies working to develop best practices, conduct research, an...governanceai-safetypolicycoordination+2Source ↗

Phase 2: Expansion (Months 12-24)

• Deploy independent behavioral evaluation infrastructure
• Establish international information sharing protocols
• Create social impact monitoring dashboards
• Implement first-generation tripwire responses

Phase 3: Optimization (Months 24-36)

• Refine detection thresholds based on empirical data
• Expand monitoring to emerging risk categories
• Develop adversarial-robust evaluation methods
• Scale international coordination mechanisms

Key Uncertainties and Cruxes

Detection Feasibility

Response Effectiveness

The effectiveness of predetermined responses to warning signs remains highly uncertain, with limited empirical evidence about what interventions successfully mitigate emerging AI risks.

Response credibility: Pre-committed responses may not be honored when economic or competitive pressure intensifies. Historical precedents from climate change and financial regulation suggest that advance commitments often weaken at decision points.

Intervention effectiveness: Most proposed interventions (deployment pauses, additional safety research, policy responses) lack empirical validation for their ability to reduce AI risks. The field relies heavily on theoretical arguments about intervention effectiveness.

Coordination sustainability: Multi-stakeholder coordination for monitoring and response faces collective action problems that may intensify as economic stakes grow and geopolitical tensions increase.

Current State and Trajectory

Monitoring Infrastructure Development

Several initiatives are establishing components of the warning signs framework, but coverage remains fragmentary and uncoordinated.

Government initiatives: The UK AI Safety Institute↗🏛️ government★★★★☆UK GovernmentAI Safety Institute - GOV.UKThis is the official UK government hub for AI safety policy and research; important for tracking state-level institutional responses to frontier AI risks and international safety coordination efforts.The UK AI Safety Institute (recently rebranded as the AI Security Institute) is a government body under the Department for Science, Innovation and Technology focused on minimizi...ai-safetygovernancepolicyevaluation+4Source ↗ and proposed US AI Safety Institute↗🏛️ government★★★★★NISTUS AI Safety InstituteNIST is a key U.S. government institution for AI safety standardization; its AI RMF and AI Safety Institute work are frequently referenced in AI governance and technical safety discussions.NIST is the U.S. national metrology and standards institute, playing a central role in AI safety through the AI Risk Management Framework (AI RMF) and hosting the U.S. AI Safety...ai-safetygovernancepolicyevaluation+4Source ↗ represent significant steps toward independent evaluation capacity. However, both organizations are resource-constrained and lack authority for mandatory reporting or response coordination.

Industry self-regulation: Anthropic's Responsible Scaling Policy↗🔗 web★★★★☆AnthropicResponsible Scaling PolicyThis is Anthropic's foundational policy document establishing how it gates deployment of increasingly capable models; a key reference for understanding industry-led AI governance frameworks and voluntary safety commitments.Anthropic introduces its Responsible Scaling Policy (RSP), a framework of technical and organizational protocols for managing catastrophic risks as AI systems become more capabl...governancepolicyai-safetycapabilities+6Source ↗ and OpenAI's Preparedness Framework↗🔗 web★★★★☆OpenAIOpenAI's Preparedness FrameworkThis is OpenAI's official internal risk governance document released publicly in late 2023; it is a key reference for understanding how a leading AI lab operationalizes safety thresholds and pre-deployment evaluation requirements for frontier models.OpenAI's Preparedness Framework outlines a systematic approach to tracking, evaluating, and mitigating catastrophic risks from frontier AI models. It establishes risk categories...ai-safetygovernanceevaluationred-teaming+6Source ↗ include elements of warning signs monitoring and tripwire responses. However, these commitments are voluntary, uncoordinated across companies, and lack external verification.

Academic research: Organizations like METR↗🔗 web★★★★☆METRMETR: Model Evaluation and Threat ResearchMETR is a leading third-party AI safety evaluation organization whose work on autonomous capability benchmarks and catastrophic risk assessments directly informs AI lab safety policies and government AI governance frameworks.METR is an organization conducting research and evaluations to assess the capabilities and risks of frontier AI systems, focusing on autonomous task completion, AI self-improvem...evaluationred-teamingcapabilitiesai-safety+5Source ↗, ARC↗🔗 webAlignment Research CenterARC is one of the leading independent technical AI safety research organizations; its evaluations work spun out as METR, and it remains influential in shaping how frontier labs approach pre-deployment safety assessments.The Alignment Research Center (ARC) is a non-profit research organization focused on technical AI alignment and safety research. ARC works on understanding and addressing risks ...ai-safetyalignmenttechnical-safetyinterpretability+5Source ↗, and Apollo Research↗🔗 web★★★★☆Apollo ResearchApollo Research - AI Safety Evaluation OrganizationApollo Research is a key third-party evaluator in the AI safety ecosystem, providing independent assessments of frontier models for dangerous capabilities and advising policymakers; their work on scheming evaluations is directly relevant to deceptive alignment concerns.Apollo Research is an AI safety organization focused on evaluating frontier AI systems for dangerous capabilities, particularly 'scheming' behaviors where advanced AI covertly p...ai-safetyevaluationred-teamingalignment+6Source ↗ are developing evaluation methodologies, but their access to frontier models remains limited and funding is insufficient for comprehensive monitoring.

Five-Year Trajectory Projections

Based on current trends and announced initiatives, the warning signs monitoring landscape in 2029 will likely feature:

The most likely outcome is partial progress on monitoring infrastructure without commensurate development of governance systems for response. This creates the dangerous possibility of detecting warning signs without capacity for effective action.

Comparative Analysis

Historical Precedents

The climate change analogy is particularly instructive: highly sophisticated monitoring systems have provided increasingly accurate warnings about risks, but institutional failures have prevented adequate response despite clear signals.

Other Risk Domains

AI warning signs monitoring can learn from more mature risk assessment frameworks:

• Financial systemic risk: Federal Reserve↗🏛️ governmentFederal Reserve System - Official WebsiteThis resource appears tagged as a monitoring/early-warning tripwire reference, suggesting it may be used to track macroeconomic indicators or financial system signals as part of an AI risk monitoring framework rather than for direct AI safety content.The Federal Reserve is the central bank of the United States, responsible for monetary policy, financial system stability, and banking regulation. It provides data, research, an...governancepolicymonitoringdeployment+1Source ↗ stress testing provides model for mandatory capability evaluation
• Cybersecurity threat detection: CISA↗🏛️ government★★★★☆CISACISA - Cybersecurity and Infrastructure Security AgencyCISA is a key U.S. government stakeholder in AI security policy; its guidelines on secure AI deployment and critical infrastructure protection are relevant to AI safety governance discussions, though this is a general homepage rather than a specific AI safety resource.CISA is the U.S. federal agency responsible for cybersecurity and critical infrastructure protection. It coordinates national efforts to defend against cyber threats, shares thr...governancepolicydeploymentcoordination+2Source ↗ information sharing demonstrates feasibility of coordinated monitoring
• Public health surveillance: CDC↗🏛️ governmentCenters for Disease Control and PreventionThe CDC homepage is included as a reference for biosecurity and public health monitoring frameworks; it is tangentially relevant to AI safety through biosecurity governance and as an institutional model for early-warning and tripwire systems.The CDC is the United States' primary federal public health agency, responsible for disease surveillance, outbreak response, health promotion, and biosafety standards. It serves...governancemonitoringbiosecuritypolicy+5Source ↗ disease monitoring shows real-time tracking at scale
• Nuclear safety: Nuclear Regulatory Commission↗🏛️ governmentNuclear Regulatory CommissionThe NRC is often cited as a governance analogy in AI safety discussions, illustrating how independent technical regulators can oversee high-risk technologies; relevant for comparative institutional design and AI governance research.The U.S. Nuclear Regulatory Commission (NRC) is the federal agency responsible for regulating civilian nuclear power plants and nuclear materials to protect public health, safet...governancepolicymonitoringregulation+3Source ↗ provides precedent for licensing with safety milestones

Expert Perspectives

Leading researchers emphasize different aspects of warning signs frameworks based on their risk models and expertise areas.

Dario AmodeiPersonDario AmodeiComprehensive biographical profile of Anthropic CEO Dario Amodei documenting his competitive safety development philosophy, 10-25% catastrophic risk estimate, 2026-2030 AGI timeline, and Constituti...Quality: 41/100 (Anthropic CEO) has argued that "responsible scaling policies must define concrete capability thresholds that trigger safety requirements," emphasizing the need for predetermined responses rather than ad hoc decision-making. Anthropic's approach focuses on creating "if-then" commitments that remove discretion at evaluation points.

Dan HendrycksPersonDan HendrycksComprehensive reference biography of Dan Hendrycks (CAIS director), covering his academic career (GELU, MMLU, OOD detection), CAIS founding and funding (including $6.5M FTX, Open Philanthropy/Coeff...Quality: 19/100 (Center for AI Safety) advocates for "AI safety benchmarks that measure existential risk-relevant capabilities," arguing that current evaluation focused on helpfulness misses the most concerning capabilities. His work emphasizes the importance of red-teaming and adversarial evaluation.

Geoffrey HintonPersonGeoffrey HintonComprehensive biographical profile of Geoffrey Hinton documenting his 2023 shift from AI pioneer to safety advocate, estimating 10-20% extinction risk in 5-20 years. Covers his media strategy, poli...Quality: 42/100 has warned that "we may not get warning signs" for the most dangerous AI capabilities, expressing skepticism about detection-based approaches. This perspective emphasizes the importance of proactive measures rather than reactive monitoring.

Stuart RussellPersonStuart RussellStuart Russell (born 1962) is a British computer scientist and UC Berkeley professor who co-authored the dominant AI textbook 'Artificial Intelligence: A Modern Approach' (used in over 1,500 univer...Quality: 30/100 argues for "rigorous testing before deployment" with emphasis on worst-case scenario evaluation rather than average-case performance metrics, highlighting the difficulty of detecting rare but catastrophic behaviors.

Sources & Resources

Academic Research

Policy and Governance

Industry Frameworks

Monitoring Organizations

International Coordination