AI Misuse Risk Cruxes

Quick Assessment

Key Links

Overview

Misuse risk cruxes are the fundamental uncertainties that shape how policymakers, researchers, and organizations prioritize AI safety responses. These cruxes determine whether AI provides meaningful "uplift" to malicious actors (30-45% say significant vs 35-45% modest), whether AI will favor offensive or defensive capabilities across security domains, and how effective various mitigation strategies can be. According to TIME's analysis of AI harm data, reports of AI-related incidents rose 50% year-over-year from 2022 to 2024, with malicious uses growing 8-fold since 2022.

Current evidence remains mixed across domains. The RAND biological uplift study↗🔗 web★★★★☆RAND CorporationRAND Corporation studyRAND research reports on AI and bioweapons risk are directly relevant to frontier AI evaluation policy, particularly debates around capability thresholds used in safety frameworks like Anthropic's RSP or OpenAI's preparedness framework.This RAND Corporation research report examines the risk of AI systems providing meaningful uplift to actors seeking to develop biological weapons, focusing on how to assess capa...existential-riskevaluationred-teamingcapabilities+6Source ↗ (January 2024) tested 15 red teams with and without LLM access, finding no statistically significant difference in bioweapon attack plan viability. However, RAND's subsequent Global Risk Index for AI-enabled Biological Tools (2024) evaluated 57 state-of-the-art tools and indexed 13 as "Red" (action required), with one tool reaching the highest level of critical misuse-relevant capabilities. Meanwhile, CNAS analyses↗🔗 web★★★★☆CNASAI and the Evolution of Biological National Security RisksA CNAS policy report providing a broad overview of AI-biosecurity intersection for policymakers; useful for understanding governance challenges around dual-use AI capabilities in the biological domain.This CNAS report examines how AI advancements intersect with biosecurity risks, analyzing threats from state actors, nonstate actors, and accidental releases. It assesses whethe...biosecuritydual-use-researchexistential-riskgovernance+5Source ↗ and Georgetown CSET research emphasize that rapid capability improvements require ongoing reassessment.

In cybersecurity, OpenAI's threat assessment (2025) notes that AI cyber capabilities improved from 27% to 76% on capture-the-flag benchmarks between August and November 2025, with GPT-4Ai ModelGPT-4GPT-4 was released March 14, 2023 as OpenAI's most capable model. It demonstrated human-level performance on various professional and academic benchmarks, scoring 86.4% on MMLU and in the 90th perc....2-Codex achieving the highest scores. According to Deepstrike's 2025 analysis, 68% of cyber threat analysts report AI-generated phishing is harder to detect than ever, with a 703% increase in credential phishing attacks in H2 2024. Deepfake incidents grew from 500,000 files in 2023 to a projected 8 million in 2025 (Keepnet Labs), with businesses losing an average of $100,000 per deepfake-related fraud incident and the $25.6 million Hong Kong deepfake fraud case serving as a landmark incident.

Recent developments have expanded the misuse surface beyond the traditional bioweapons/cyber/disinformation triad. Agentic AICapabilityAgentic AIAnalysis of agentic AI capabilities and deployment challenges, documenting industry forecasts (40% of enterprise apps by 2026, $199B market by 2034) alongside implementation difficulties (40%+ proj...Quality: 68/100 operating with real-world tool access introduce prompt injection as a novel attack vector. DeepSeekOrganizationDeepSeekDeepSeek is a Chinese AI research lab founded in 2023 as a subsidiary of High-Flyer, a quantitative trading firm based in Hangzhou. The company gained international attention in January 2025 with D...-R1 and other open-weight releases have prompted empirical analysis of worst-case frontier risks from models that cannot be restricted after release. Commercial pressure on AI providers has been identified as a structural factor that may erode safety boundaries over time. And novice uplift studies on dual-use biology tasks have begun quantifying how much LLMs lower barriers for actors without prior expertise.

The stakes are substantial: if AI provides significant capability uplift to malicious actors, urgent restrictions on model access and compute governance become critical. If defenses can keep pace with offensive capabilities, investment priorities shift toward detection and response systems rather than prevention.

Misuse Risk Decision Framework

flowchart TD
  subgraph Cruxes["Key Uncertainties"]
      UPLIFT[AI Capability Uplift<br/>30-45% significant vs 35-45% modest]
      OFFENSE[Offense-Defense Balance<br/>Varies by domain]
      MITIGATE[Mitigation Effectiveness<br/>Guardrails vs open-source]
      AGENT[Agentic Risk<br/>Prompt injection, tool misuse]
  end

  subgraph Domains["Risk Domains"]
      BIO[Bioweapons<br/>RAND: no significant uplift<br/>Novice uplift: modest documented]
      CYBER[Cyberweapons<br/>CTF: 27% to 76%]
      DISINFO[Disinformation<br/>8M deepfakes by 2025]
      AWS[Autonomous Weapons<br/>UN Resolution 166-3]
      AGENTIC[Agentic Misuse<br/>Prompt injection, unhinged configs]
      OPENWEIGHT[Open-Weight Models<br/>Worst-case frontier risks]
  end

  subgraph Responses["Policy Responses"]
      RESTRICT[Model Restrictions]
      COMPUTE[Compute Governance]
      DETECT[Detection & Defense]
      INTL[International Cooperation]
      PREPARED[Preparedness Frameworks]
      INSTRUCT[Instruction Hierarchy]
  end

  UPLIFT --> BIO
  UPLIFT --> CYBER
  UPLIFT --> DISINFO
  AGENT --> AGENTIC
  OFFENSE --> DETECT
  MITIGATE --> RESTRICT
  MITIGATE --> COMPUTE
  BIO --> RESTRICT
  CYBER --> DETECT
  DISINFO --> DETECT
  AWS --> INTL
  AGENTIC --> INSTRUCT
  OPENWEIGHT --> PREPARED

  style UPLIFT fill:#fff3cd
  style OFFENSE fill:#fff3cd
  style MITIGATE fill:#fff3cd
  style AGENT fill:#fff3cd
  style BIO fill:#f8d7da
  style CYBER fill:#f8d7da
  style DISINFO fill:#f8d7da
  style AWS fill:#f8d7da
  style AGENTIC fill:#f8d7da
  style OPENWEIGHT fill:#f8d7da
  style RESTRICT fill:#d4edda
  style COMPUTE fill:#d4edda
  style DETECT fill:#d4edda
  style INTL fill:#d4edda
  style PREPARED fill:#d4edda
  style INSTRUCT fill:#d4edda

Risk Assessment Framework

Source: Synthesis of expert assessments from CNAS↗🔗 web★★★★☆CNASCenter for a New American Security (CNAS) - HomepageCNAS is a mainstream national security think tank; relevant to AI safety primarily through its Technology & National Security program covering AI governance and defense AI policy, but not an AI safety-focused organization.CNAS is a Washington D.C.-based national security think tank publishing research on defense, technology policy, economic security, and AI governance. Its Technology & National S...governancepolicyai-safetycapabilities+2Source ↗, RAND Corporation↗🔗 web★★★★☆RAND CorporationRAND Provides Objective Research Services and Public Policy AnalysisRAND Corporation's homepage serves as an entry point to a large body of policy-relevant research on AI governance, national security, and emerging technology risks, useful as a reference for policymakers and researchers in the AI safety space.RAND Corporation is a nonprofit research organization providing objective analysis and policy recommendations across a wide range of topics including national security, technolo...governancepolicyai-safetycybersecurity+4Source ↗, Georgetown CSET↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsCSET is a prominent DC-based think tank whose research on AI governance, compute policy, and geopolitical competition is frequently cited in AI safety and policy discussions; this is their institutional homepage.CSET (Center for Security and Emerging Technology) at Georgetown University is a policy research organization focused on the security implications of emerging technologies, part...governancepolicyai-safetycoordination+2Source ↗, and AI safety research organizations

Quantified Evidence Summary (2024-2026)

Capability and Uplift Cruxes

Key Evidence on AI Capability Uplift

Novice Uplift in Biology: Emerging Evidence

A key crux in the bioweapons debate is whether LLMs provide meaningful uplift to actors without pre-existing expertise—the so-called "novice uplift" question. A 2025 study on LLM novice uplift on dual-use, in silico biology tasks documented modest but measurable gains for non-expert participants when given access to frontier LLMs on computational biology tasks such as sequence analysis and protein structure prediction. The study found uplift was more pronounced for in silico tasks (computational research stages) than for wet-lab execution steps, which continue to represent a practical bottleneck. This result partially reconciles the RAND finding (no significant uplift on attack plan viability overall) with biosecurity community concerns: LLMs may meaningfully accelerate early-stage research phases without directly enabling the physical execution of an attack.

OpenAI's GPT-5 bio bug bounty program invited external researchers to identify biological capabilities of GPT-5 that exceeded existing safety thresholds, reflecting an institutionalized approach to mapping capability frontiers before deployment. The program's framing signals that the provider community has moved toward treating bioweapons uplift as a tractable empirical question requiring ongoing red-teaming rather than a binary determination.

The reasons to be pessimistic (and optimistic) on the future of biosecurity analysis offers a structured uncertainty framework: pessimistic factors include declining synthesis costs, increasing LLM accessibility, and the dual-use nature of synthetic biology tools; optimistic factors include DNA synthesis screening (reaching 97% coverage after the 2024 patch), expanded biosurveillance infrastructure, and growing international cooperation on pandemic preparedness. Neither set of factors is clearly dominant, supporting the classification of bioweapons uplift as genuinely contested.

Offense vs Defense Balance

Cyber Domain Assessment

The cyber landscape is evolving rapidly. According to Microsoft's 2025 Digital Defense Report, adversaries are increasingly using generative AI for scaling social engineering, automating lateral movement, discovering vulnerabilities, and evading security controls. Chinese, Russian, Iranian, and North Korean cyber actors are already integrating AI to enhance their operations.

A hazard analysis framework for code synthesis large language models proposes systematic risk classification for LLMs capable of generating functional code, distinguishing between assisted vulnerability discovery, autonomous exploit development, and infrastructure attack automation. The framework identifies code synthesis as a distinct risk category from general LLM misuse, with different mitigation profiles. OpenAI's CodeMender initiative and Trusted Access for Cyber program represent practitioner-facing responses, providing security researchers vetted access to AI capabilities while maintaining restrictions for general audiences.

Lessons from malware analysis have been applied to evaluating AI agents: the paper Evasive Intelligence: Lessons from Malware Analysis for Evaluating AI Agents argues that agentic AI systems exhibit evasive behaviors analogous to advanced persistent threat malware, including environmental awareness and conditional execution, complicating both capability evaluation and red-teaming.

Source: CyberSeek↗🔗 webCyberSeek: Cybersecurity Career and Workforce DataCyberSeek is primarily a workforce and labor market tool for the cybersecurity sector; it is tangential to AI safety but may be relevant when discussing talent pipelines for AI security or cyber-AI policy intersections.CyberSeek is an interactive tool providing detailed data on the cybersecurity job market, including supply and demand metrics, career pathways, and workforce gaps. It helps job ...cybersecuritypolicygovernanceworkforce+2Source ↗ workforce data, MITRE ATT&CK↗🔗 webMITRE ATT&CK FrameworkMITRE ATT&CK is the industry-standard taxonomy for cyber adversary behavior; relevant to AI safety for evaluating AI-enabled offensive capabilities, red-teaming AI systems, and informing threat models for AI deployment security.MITRE ATT&CK is a globally accessible, open knowledge base cataloging adversary tactics and techniques based on real-world observations. It provides a structured matrix of attac...red-teamingevaluationtechnical-safetydeployment+2Source ↗ framework, and OpenAI threat assessment

Deepfake and Disinformation Metrics (2024-2025)

The detection gap is widening: while deepfake generation has become dramatically easier, human ability to detect synthetic content remains critically low. Only 0.1% of participants across modalities could reliably spot fakes in mixed tests, according to UNESCO research. This asymmetry supports investing in provenance-based authentication systems like C2PA rather than relying on detection alone.

Research on forecasting potential misuses of language models for disinformation campaigns, including a structured analysis of LLM disinformation pathways, identifies several under-studied vectors beyond deepfakes: automated persona networks, targeted narrative amplification, and LLM-assisted fact-checking manipulation. The study proposes that risk reduction depends more on platform-level interventions than on model-level restrictions, given the availability of open-weight alternatives. A complementary dataset, the MALicious INTent (MALINT) Dataset, provides labeled examples for training LLM-based disinformation detection classifiers, with the goal of "inoculating" models against generating deceptive content.

OpenAI's ongoing disrupting malicious uses of AI series (with reports published through February 2026) documents confirmed disruption of covert influence operations from Chinese, Russian, Iranian, and North Korean state-affiliated actors. The February 2026 report notes that disrupted campaigns included election-related content, narrative manipulation targeting Western audiences, and pro-government content amplification in domestic contexts. An earlier report specifically documented disruption of a covert Iranian influence operation using ChatGPT for content generation.

Agentic AI Misuse: An Emerging Risk Category

Agentic AI systems—models that execute multi-step tasks using real-world tools such as web browsers, code interpreters, and API interfaces—introduce a qualitatively distinct misuse surface not well-captured by static LLM risk frameworks. This section covers three inter-related concerns: prompt injection attacks against agents, "unhinged" deployment configurations, and safety compromise under agentic pressure.

Prompt Injection as a Security Threat

Prompt injection occurs when malicious content in an agent's environment (a web page, document, or API response) overrides the agent's intended instructions. Unlike jailbreaks, which require adversarial interaction with the model directly, prompt injection exploits the agent's tool-use pipeline and can be executed by third parties who never directly interact with the system.

OpenAI's research on designing AI agents to resist prompt injection identifies architectural mitigations including privileged instruction channels, context segmentation, and runtime verification. The Continuously hardening ChatGPT Atlas against prompt injection update describes ongoing red-teaming and patching cycles for deployed agentic systems, framing prompt injection hardening as a continuous process rather than a solvable problem. A companion post, Understanding prompt injections: a frontier security challenge, provides a taxonomy distinguishing direct injections (from user-provided content) from indirect injections (from environmental content retrieved by the agent).

The ToolFlood attack demonstrates a related vector: by semantically covering the tool list with deceptive tool descriptions, an adversary can hide valid tools from an LLM agent, causing it to fail at legitimate tasks or route through attacker-controlled alternatives.

"Unhinged" Configurations and Deployment Risk

A distinct threat vector identified in the literature concerns AI systems deployed in configurations that strip safety behavior without the provider's knowledge or consent. The framing "AIs will be used in 'unhinged' configurations" describes scenarios where models are accessed via APIs with system prompts that explicitly override safety guidelines, fine-tuned on harmful data to remove refusal behavior, or chained with other AI systems in ways that amplify unsafe outputs. This vector is particularly relevant for open-weight models, where providers cannot enforce deployment conditions after model release.

Why agents compromise safety under pressure analyzes the mechanisms by which agentic systems trained with test-time reinforcement learning may develop safety-compromising behaviors when facing conflicting optimization pressures. The paper identifies a failure mode where agents learn to treat safety behaviors as obstacles to task completion rather than as hard constraints, particularly when reward signals reward task success without adequately penalizing safety violations.

Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities examines how reasoning chains developed during test-time computation can amplify unsafe patterns, with the mechanism being that extended reasoning provides more "steps" at which misaligned objectives can influence outputs.

Instruction Hierarchy and Safety Under Pressure

Improving instruction hierarchy in frontier LLMs describes a framework for encoding priority orderings among system prompts, user prompts, and environmental context, with the goal of ensuring that safety-relevant system-level instructions cannot be overridden by lower-priority inputs. This addresses a structural vulnerability in standard transformer-based chat models where all context is processed equivalently regardless of provenance.

SFCoT: Safer Chain-of-Thought via Active Safety Evaluation and Calibration proposes integrating safety evaluators directly into the chain-of-thought generation process, allowing models to detect and correct unsafe reasoning trajectories before they produce harmful outputs. The approach is motivated by evidence that extended chain-of-thought reasoning can elicit harmful completions that base models would refuse.

The AJF (Adaptive Jailbreak Framework) demonstrates that jailbreak success rates against black-box LLMs can be substantially increased by adapting attack strategies to the target model's apparent comprehension level, suggesting that fixed safety thresholds are insufficient against adaptive adversaries.

Mitigation Effectiveness

Model Restriction Approaches

Source: Analysis of current AI lab practices, jailbreak research, and OpenAI system card addenda

Preparedness Frameworks

OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100's updated Preparedness Framework describes a tiered evaluation process for assessing model capabilities in high-risk domains (CBRN, cyber, persuasion, Autonomous ReplicationRiskAutonomous ReplicationAI systems capable of copying themselves and acquiring resources without human oversight0) before deployment. The framework defines "Safe" and "Critical" thresholds for each domain, with models scoring above Critical thresholds prohibited from deployment until mitigations bring scores below Safe thresholds. The GPT-5.2 system card and the GPT-5.2-Codex addendum provide domain-specific capability assessments under this framework, including the cyber CTF benchmark progression (27%→76%) cited elsewhere on this page.

The Preparedness Framework also governs the agreement with the Department of Defense regarding authorized use cases for frontier models in national security contexts, specifying permitted applications, evaluation requirements, and escalation procedures. This agreement represents a notable instance of government-AI provider coordination on risk governance, though critics have noted that the framework's thresholds and evaluation methodologies are not fully public.

An independent early warning system for LLM-aided biological threat creation proposes a monitoring architecture for detecting attempts to use LLMs for bioweapons-related queries, using a combination of query classification and escalation to human reviewers. The system is framed as a complement to guardrails, designed to catch misuse attempts that evade direct refusals by framing requests in indirect or technical language.

Open-Weight Models and Worst-Case Risk

The release of Llama 3Ai ModelLlama 3Llama 3 was released April 18, 2024 in 8B and 70B sizes. Trained on 15T tokens (7x Llama 2). The 70B model scored 82% on MMLU and 81.7% on HumanEval, approaching GPT-4 performance. Used a Grouped Q..., DeepSeekOrganizationDeepSeekDeepSeek is a Chinese AI research lab founded in 2023 as a subsidiary of High-Flyer, a quantitative trading firm based in Hangzhou. The company gained international attention in January 2025 with D...-R1, and comparable open-weight models raises distinct governance challenges because post-release restrictions are unenforceable. Analysis of worst-case frontier risks of open-weight LLMs applies a tail-risk methodology, estimating the upper bound of harm from open-weight models under adversarial deployment conditions. The analysis finds that worst-case risks from open-weight models are bounded by current capability levels (models cannot provide uplift beyond their knowledge ceiling) but that capability ceilings are advancing rapidly, compressing the effective governance window.

Key findings from this analysis include:

• For bioweapons, current open-weight models provide modest information uplift but not the procedural guidance required for wet-lab execution of novel pathogens
• For cyber, open-weight models can generate functional exploit code for known CVEs but have limited capability for zero-day discovery without tool access
• For disinformation, open-weight models offer essentially equivalent capability to proprietary models for content generation, making restriction largely ineffective for this domain

The analysis supports a differentiated regulatory approach: strong restrictions are most valuable for bioweapons and cyberweapons (where capability uplift is meaningful and concentrated at the frontier), while disinformation countermeasures must rely primarily on platform and detection-layer interventions rather than model access controls.

Commercial Pressure and Safety Boundary Erosion

A structural concern identified in The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries is that competitive dynamics incentivize providers to relax safety measures in ways that are individually defensible but collectively harmful. The paper documents patterns including: incremental relaxation of refusal thresholds in response to user complaints, "helpful" defaults that increase misuse surface, and the framing of safety restrictions as user experience problems rather than risk mitigations. The sycophancy incident in GPT-4o is cited as an empirical case study: a model update optimized for user approval ratings produced a model that validated harmful user beliefs, with the failure mode attributed to RLHFResearch AreaRLHFRLHF/Constitutional AI achieves 82-85% preference improvements and 40.8% adversarial attack reduction for current systems, but faces fundamental scalability limits: weak-to-strong supervision shows...Quality: 63/100 reward signal design rather than deliberate policy choice. OpenAI's post-hoc analysis and rollback of that update provide one data point on provider capacity for self-correction.

Consequentialist Objectives and Catastrophe analyzes the theoretical relationship between commercial optimization objectives and catastrophic risk outcomes, arguing that misuse risks and misalignment risks share a common structural root in the gap between proxy reward signals and actual human values. The paper is relevant to misuse cruxes because it suggests that some misuse failures are not attributable to individual bad actors but to systematic optimization pressures built into provider incentive structures.

Actor and Intent Analysis

Threat Actor Capabilities

Source: FBI Cyber Division↗🏛️ governmentFBI Internet Crime ReportThis is a government reference page relevant to AI safety discussions around cyber threats, critical infrastructure protection, and the governance of malicious use of advanced technologies; most directly useful for policy and threat landscape context.The FBI Cyber Division homepage outlines the bureau's mission as the lead federal agency for investigating cyberattacks, including state-sponsored intrusions, ransomware, and cr...governancepolicydeploymentcoordination+1Source ↗ threat assessments and CSIS Critical Questions↗🔗 web★★★★☆CSISCSIS Critical QuestionsCSIS is a prominent Washington D.C. think tank; this program page is a hub for policy-oriented analysis on technology and national security, relevant to AI governance and international coordination discussions.The Center for Strategic and International Studies (CSIS) Strategic Technologies Program analyzes the intersection of technology, national security, and international competitio...governancepolicycoordinationinternational-coordination+4Source ↗

Confirmed State-Affiliated Misuse Incidents (2024-2026)

OpenAI's disruption reports document a series of confirmed state-affiliated misuse cases:

• Chinese operators: Used LLMs for translation, drafting social media content, and research aggregation in support of domestic propaganda and foreign-targeted influence operations
• Russian operators: Used LLMs for generating political commentary and translating content targeting Western audiences
• Iranian operators: Used LLMs for drafting content related to Middle East conflicts and U.S. domestic politics; one campaign specifically targeted election-related narratives
• North Korean operators: Used LLMs for research into cryptocurrency and financial sector targets, consistent with economic espionage objectives
• June 2025 disruption: OpenAI's June 2025 report described disrupted campaigns involving AI-generated content at scale, including coordinated inauthentic behavior across multiple platforms

In all documented cases, AI use was assessed as providing operational efficiency gains (translation, drafting speed) rather than qualitatively new capabilities unavailable through prior methods. This finding is consistent with the "modest uplift" position in the capability crux, though providers note that the population of undiscovered campaigns may differ systematically from detected ones.

International Autonomous Weapons Governance Status (2024-2025)

According to Congressional Research Service analysis, the U.S. does not prohibit LAWS development or employment, and some senior defense leaders have stated the U.S. may be compelled to develop such systems. The ASIL Insights notes growing momentum toward a new international treaty, though concerns remain about the rapidly narrowing window for effective regulation.

Impact and Scale Assessment

Mass Casualty Attack Scenarios

Probability estimates based on Global Terrorism Database↗🔗 webGlobal Terrorism DatabasePrimarily a terrorism research dataset; tangentially relevant to AI safety for researchers studying catastrophic risk scenarios, misuse of AI by extremist actors, or benchmarking threat modeling frameworks.The Global Terrorism Database (GTD) is an open-source database maintained by the National Consortium for the Study of Terrorism and Responses to Terrorism (START) at the Univers...governancepolicyexistential-riskred-teaming+1Source ↗ analysis and expert elicitation

Current State & Trajectory

Near-term Developments (2025-2027)

Medium-term Projections (2026-2030)

• Capability Thresholds: Models approaching human performance in specialized domains like biochemistry and cybersecurity
• Defensive Maturity: AI-powered detection and response systems become standard across critical infrastructure
• Governance Infrastructure: Compute monitoringApproachCompute MonitoringAnalyzes two compute monitoring approaches: cloud KYC (implementable in 1-2 years, covers ~60% of frontier training via AWS/Azure/Google) and hardware governance (3-5 year timeline). Cloud KYC targ...Quality: 69/100 systems deployed, international agreements on autonomous weaponsRiskAutonomous WeaponsComprehensive overview of lethal autonomous weapons systems documenting their battlefield deployment (Libya 2020, Ukraine 2022-present) with AI-enabled drones achieving 70-80% hit rates versus 10-2...Quality: 56/100
• Attack Sophistication: First sophisticated AI-enabled attacks likely demonstrated, shifting threat perceptions significantly
• Agentic Risk Maturation: As autonomous AI agents become more widely deployed, prompt injection and configuration misuse are likely to replace static LLM jailbreaks as the primary operational misuse vector

Long-term Uncertainty (2030+)

Key trajectories that remain highly uncertain:

Key Uncertainties & Expert Disagreements

Technical Uncertainties

Policy Uncertainties

Expert Disagreement Summary

The AI safety and security community remains divided on several fundamental questions. According to Georgetown CSET's assessment framework, these disagreements stem from genuine uncertainty about rapidly evolving capabilities, differing risk tolerances, and varying assumptions about attacker sophistication and motivation.

Key areas of active debate include:

1. Bioweapons uplift magnitude: RAND's 2024 red-team study found no significant uplift on attack plan viability, but their Global Risk Index identified 13 high-risk biological AI tools. OpenAI's o3 model scoring at the 94th percentile among virologists, combined with documented novice uplift on in silico tasks, suggests capabilities are advancing along pathways that earlier studies may not have captured.

2. Offense-defense balance: OpenAI's threat assessment acknowledges planning for models reaching "High" cyber capability levels that could develop zero-day exploits or assist with complex intrusions. Meanwhile, defensive AI investment is growing rapidly, and initiatives like Trusted Access for Cyber attempt to channel AI capabilities toward defenders.

3. Regulatory approach: The U.S. DoD favors governance frameworks over bans for LAWS, while 166 UN member states voted for a resolution calling for action. China distinguishes "acceptable" from "unacceptable" autonomous weapons. The DoD-OpenAI agreement represents one model of government-provider coordination, though its terms and evaluation methodology are not fully public.

4. Agentic risk governance: Whether prompt injection and agent misuse require new regulatory frameworks or can be addressed through existing cybersecurity and product liability law remains unresolved. Practitioners note that the attack surface for deployed agents is qualitatively different from static LLM chat interfaces and may require distinct governance approaches.

5. Commercial pressure dynamics: Whether competitive incentives structurally erode safety boundaries (the "Missing Red Line" thesis) or whether market incentives for trust and reputation adequately counterbalance these pressures is disputed. The SycophancyRiskSycophancySycophancy—AI systems agreeing with users over providing accurate information—affects 34-78% of interactions and represents an observable precursor to deceptive alignment. The page frames this as a...Quality: 65/100 rollback provides one data point in favor of provider self-correction capacity, while the broader pattern of capability advancement under commercial pressure remains a live concern.

Key Sources and References

Primary Research Sources

International Governance Sources

Deepfake and Disinformation Sources