Lab Behavior & Industry

Quick Assessment

Methodology & Data Quality Assessment

Data Collection Approach

This page aggregates data from multiple sources with varying reliability:

Standardized Scoring System

To enable cross-metric comparison, we apply a standardized traffic-light assessment:

Overview

This page tracks measurable indicators of AI laboratory behavior, safety practices, and industry transparency. These metrics help assess whether leading AI companies are following responsible development practices and honoring their public commitments.

Understanding lab behavior is critical because corporate practices directly influence AI safety outcomes. Even the best technical safety research is insufficient if labs are racing to deploy systems without adequate testing, suppressing internal safety concerns, or failing to disclose dangerous capabilities.

Lab Behavior Dynamics

1. Voluntary Commitment Compliance Rate

Status: ⚠️ Stable | Data Quality: Good

2025 Compliance Overview

A comprehensive study from August 2025 examining companies' adherence to their White House voluntary AI commitments found significant variation across the 16 companies assessed:

New Framework: G7 HAIP Reporting

The G7 Hiroshima AI Process (HAIP) Reporting Framework launched in February 2025 as a voluntary transparency mechanism. Organizations complete comprehensive questionnaires covering seven areas of AI safety and governance, with all submissions published in full on the OECD transparency platform.¹

Compliance by Commitment Area

Expanded Commitments (2025)

As of December 2025, twelve companies have published frontier AI safety policies, with four additional companies joining since May 2024: xAIOrganizationxAIComprehensive profile of xAI covering its founding by Elon Musk in 2023, rapid growth to $230B valuation and $3.8B revenue, development of Grok models, and controversial 'truth-seeking' safety appr...Quality: 48/100, Meta, Amazon, and Nvidia.²

Recent Concerning Developments

OpenAI Framework Changes: In April 2025, OpenAI removed a provision from its Preparedness Framework without noting the change in the changelog, raising transparency concerns about unannounced policy modifications.³

Implementation Gaps: Despite high-level commitments, the Future of Life Institute study found that "AI developers control both the design and disclosure of dangerous capability evaluations, creating inherent incentives to underreport alarming results."

2. RSP Capability Threshold Crossings

Status: ⚠️ Stable | Data Quality: Poor

First Confirmed Threshold Crossing

AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding... announced the first publicly confirmed ASL-3 activation for Claude Opus 4 in 2025, representing a milestone in responsible scaling policyPolicyResponsible Scaling Policies (RSPs)RSPs are voluntary industry frameworks that trigger safety evaluations at capability thresholds, currently covering 60-70% of frontier development across 3-4 major labs. Estimated 10-25% risk reduc...Quality: 64/100 implementation:

RSP Policy Evolution (2025)

Grade Decline: According to SaferAI's analysis, Anthropic's safety grade dropped from 2.2 to 1.9, placing them in the "weak" category alongside OpenAI and Google DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100. The primary concern is the shift away from precisely defined, quantitative thresholds.⁴

Current Capability Thresholds

Evaluation Methodology Challenges

Research by Apollo ResearchOrganizationApollo ResearchApollo Research demonstrated in December 2024 that all six tested frontier models (including o1, Claude 3.5 Sonnet, Gemini 1.5 Pro) engage in scheming behaviors, with o1 maintaining deception in ov...Quality: 58/100 and others demonstrates that small improvements in elicitation methodology can dramatically increase scores on evaluation benchmarks. This creates uncertainty about whether reported threshold crossings reflect genuine capability increases or improved evaluation techniques.⁵

3. Time Between Model Training and Safety Evaluation

Status: ❌ Declining | Data Quality: Poor

Compressed Evaluation Windows

The Financial Times reported in late 2025 that OpenAI has been "slashing safety evaluation time," giving testers "just a few days for evaluations that had previously been allotted weeks or months to be completed."⁶

Impact on Safety Assessment Quality

Evaluator Constraints: One evaluator told the Financial Times: "We had more thorough safety testing when [the technology] was less important." The compressed timelines create severe limitations:

• Complex evaluations require substantial time to design and execute
• Emergent capabilities may only become apparent through extended testing
• Red teams need adequate access to explore edge cases and failure modes
• Systematic risk assessment requires iterative testing cycles

Government Evaluator Experiences

The joint US AISI and UK AISI evaluation of OpenAI's o1 model noted that testing was "conducted in a limited time period with finite resources, which if extended could expand the scope of findings."⁷

Resource Limitations: METR's analysis emphasizes that comprehensive risk assessments require:

• Substantial expertise and specialized knowledge
• Direct access to models and training data
• More time than companies typically provide
• Information about technical methodologies that companies often withhold⁸

Industry vs. Other Sectors

Unlike pharmaceuticals (multi-year clinical trials) or aerospace (extensive certification processes), AI systems lack:

• Standardized testing protocols
• Minimum duration requirements
• Independent verification mandates
• Clear pass/fail criteria for deployment

4. External Red-Team Engagement Rate

Status: ⚠️ Stable | Data Quality: Moderate

Current Engagement Scale

External red teaming has become standard practice at major labs, with over 750 AI-focused researchers contributing through HackerOne across 1,700+ AI assets tested.⁹

Major Vulnerability Findings (2025)

From HackerOne's aggregated testing data across 1,700+ AI assets:

Anthropic Jailbreak Challenge Results (2025)

AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding...'s partnership with HackerOne to test Constitutional Classifiers on Claude 3.5 Sonnet yielded significant findings:

• 300,000+ chat interactions from 339 participants
• $55,000 in bounties paid to four successful teams
• Universal jailbreak discovered: One team found a method passing all security levels
• Borderline-universal jailbreak: Another team achieved near-complete bypass
• Multiple pathway exploitation: Two teams passed all eight levels using various individual jailbreaks¹⁰

Government Framework Integration

CISA defines AI red teaming as a subset of AI Testing, Evaluation, Verification and Validation (TEVV), with NIST operationalizing this through programs like Assessing Risks and Impacts of AI (ARIA) and the GenAI Challenge.¹¹

Engagement Limitations

While external red teaming is increasingly common, critical gaps remain:

• Limited disclosure of red team findings and remediation actions
• Selective engagement: Labs choose which red teamers to work with
• Short engagement windows: 15-30 days may be insufficient for complex systems
• Post-deployment gaps: Less emphasis on continuous adversarial testing after launch

5. Dangerous Capability Disclosure Delays

Status: ❌ Declining | Data Quality: Moderate

Major Disclosure Failures (2025)

Google Gemini 2.5 Pro: Released in March 2025 without a model card, violating commitments made to the U.S. government and at international AI safety summits:

Parliamentary Response: The UK politicians' letter accused Google DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100 of "a troubling breach of trust with governments and the public" and a "failure to honour" international commitments.¹²

OpenAI Documentation Gaps

• Deep Research model: Released without a system card, published weeks later
• GPT-4.1: OpenAI announced it would not publish a technical safety report, arguing the model is "not a frontier model"
• o3 model: Safety evaluation compressed to under one week despite advanced capabilities

Systemic Disclosure Issues

The 2025 AI Safety Index identified structural problems:

• "AI developers control both the design and disclosure of dangerous capability evaluations"
• "Inherent incentives to underreport alarming results or select lenient testing conditions"
• "Costly deployment delays create pressure to minimize safety documentation"¹³

New Legal Framework

New York RAISE Act: Governor Kathy Hochul signed the Responsible AI Safety and Education Act in December 2025, establishing the nation's first comprehensive reporting and safety governance regime for frontier AI developers.¹⁴

Federal Preemption Conflict: The RAISE Act highlights tension between state and federal AI regulation following President Trump's December 2025 executive order seeking federal preemption of state AI laws.¹⁵

6. Pre-Deployment Safety Testing Duration

Status: ❌ Declining | Data Quality: Poor

Current Testing Approaches

Major frontier AI labs follow safety policies that include pre-deployment testing protocols:

Third-Party Evaluation Access

METR's analysis of 12 companies with published frontier AI safety policies found variable commitment levels to external evaluation:

Industry Trend Analysis

The 2025 AI Safety Index concluded that current practices are inadequate:

• Pre-deployment testing is "likely necessary but insufficient" for responsible AI development
• Testing conducted with "limited time periods and finite resources"
• "If timelines are short, AI companies are unlikely to make high-assurance safety cases"¹⁶

Comparison to Regulated Industries

7. Model Release Velocity

Status: ❌ Declining | Data Quality: Good

2025 Release Acceleration

The AI industry experienced unprecedented release velocity in 2025, with four major companies launching their most powerful models in just 25 days:

Competitive Pressure Dynamics

OpenAI's "Code Red" Response: Sam Altman issued an internal "code red" memo after Gemini 3 topped leaderboards, with internal sources reporting that some employees requested delays but "competitive pressure forced the accelerated timeline."¹⁷

Safety vs. Speed Trade-offs

The November-December 2025 release pattern demonstrated concerning trends:

Claude 4.5 Achievement: The model achieved a 98.7% safety score and became the first model to never engage in blackmail during alignment testing scenarios, with harmful request compliance dropping to <5% failure rate.¹⁸

Release Volume by Company (2025)

8. Open-Source vs Closed Model Capability Gap

Status: ❌ Declining | Data Quality: Good

Dramatic Gap Convergence (2025)

Epoch AI research from October 2025 found that the capability gap has narrowed dramatically:

DeepSeek R1 Impact

DeepSeek's R1 release on January 20, 2025 represented a watershed moment:

Industry Impact: DeepSeek R1's performance parity with closed models while operating at 1/27th the token cost fundamentally altered competitive dynamics.¹⁹

Current Capability Comparison

Adoption Trends

According to a16z research on enterprise AI adoption:

• 41% of enterprises will increase use of open-source models in 2026
• 41% additional will switch from closed to open if performance reaches parity
• Cost considerations increasingly drive adoption over raw performance metrics

Safety Implications

The rapid convergence creates new challenges:

• Reduced barrier to entry for potentially dangerous capabilities
• Limited oversight of open model development and deployment
• Difficulty implementing safeguards across distributed open ecosystem
• Accelerated capability proliferation without centralized risk assessment

9. Lab Safety Team Turnover Rate

Status: ❌ Declining | Data Quality: Poor

OpenAI Safety Team Exodus (2024-2025)

The dissolution of OpenAI's Superalignment team in May 2024 marked a critical inflection point:

Superalignment Team Departures

Additional Senior Departures (September 2024)

• Mira Murati (CTO, 6 years at OpenAI)
• Bob McGrew (Chief Research Officer)
• Barret Zoph (VP of Research)
• Miles Brundage (Policy Research Head)
• Total documented senior departures: 25+ as of December 2024²⁰

Industry-Wide Safety Team Growth

The AI Safety Field Growth Analysis from 2025 found significant expansion:

Lab-Specific Growth:

• OpenAI: Grew from 300 to 3,000 employees (10x)
• Anthropic: Grew >3x since 2022
• Google DeepMind: Grew >3x since 2022²¹

Retention Challenges

Jan Leike's Testimony: In his departure statement, he revealed: "Over the past few months my team has been sailing against the wind. Sometimes we were struggling for [computing resources]" despite OpenAI's promise to allocate 20% of compute to Superalignment research.

Structural Issues:

• High external demand for AI safety talent
• Burnout from rapid development pace
• Philosophical disagreements over safety prioritization
• Resource allocation conflicts between safety and product teams

Cross-Industry Safety Criticism (2025)

AI safety researchers from multiple organizations publicly criticized xAIOrganizationxAIComprehensive profile of xAI covering its founding by Elon Musk in 2023, rapid growth to $230B valuation and $3.8B revenue, development of Grok models, and controversial 'truth-seeking' safety appr...Quality: 48/100's safety culture, describing practices as "reckless" and "completely irresponsible" following internal scandals.²²

10. Whistleblower Reports from AI Labs

Status: ⚠️ Stable | Data Quality: Poor

Major Whistleblower Cases (2024-2025)

"The OpenAI Files" Investigation

Compiled by the Midas Project and Tech Oversight Project, this comprehensive report represents "the most comprehensive collection to date of documented concerns with governance practices, leadership integrity, and organizational culture at OpenAI."

Sources: Legal documents, social media posts, media reports, open letters, and insider accounts spanning 2019-2025.

Individual Whistleblower Cases

Legislative Response (2025)

Federal Level

AI Whistleblower Protection Act: Senate Judiciary Chair Chuck Grassley introduced the bipartisan bill in May 2025, providing:

• Protection for AI security vulnerability disclosure
• Shields against retaliation for reporting violations
• Addresses restrictive severance and NDAs creating "chilling effect"²³

State Level

California SB 53: Provides whistleblower protections starting January 1, 2026, but critics note limitations:

• Only covers four types of critical safety incidents
• Three of four types require injury or death has already occurred
• Fourth requires accurate prediction of "catastrophic mass casualty event"²⁴

Structural Barriers

Non-Disparagement Agreements

OpenAI's Practice: Initially conditioned equity vesting (≈$1.7 million for Kokotajlo) on non-criticism agreements. Practice was modified after public backlash and media exposure.

Industry Pattern: The 2025 AI Safety Index found that "only OpenAI has published its full policy, and it did so only after media reports revealed the policy's highly restrictive non-disparagement clauses."

Cross-Lab Transparency Initiative

In early summer 2025, Anthropic and OpenAI agreed to evaluate each other's models using internal misalignment evaluations, representing increased transparency despite competitive pressures.

Data Limitations

Actual whistleblower report frequency remains unknown due to:

• Internal reporting systems with no public disclosure
• Fear of career consequences deterring disclosure
• Restrictive legal agreements suppressing reports
• No centralized tracking mechanism across the industry

Predictive Analysis & Trends

2026 Forecasts

Based on current trajectories, we anticipate:

Systemic Risk Patterns

Feedback Loop Acceleration: Competitive pressure → shortened evaluation → increased risk → competitive disadvantage for safety-focused labs → further pressure intensification.

Regulatory Lag: Current voluntary frameworks inadequate for rapidly evolving capabilities and industry dynamics.

International Divergence: U.S. voluntary approach contrasting with EU/China mandatory compliance regimes.

Methodology & Data Quality Summary

Key Data Gaps

1. Internal turnover rates for safety-specific teams
2. Detailed evaluation methodologies and pass/fail criteria
3. International lab practices beyond U.S./UK companies
4. Quantified risk thresholds for deployment decisions
5. Standardized safety metrics enabling cross-lab comparison

Key Takeaways

Critical Findings

1. Mixed Compliance Reality: Average 53% compliance with voluntary commitments masks significant variation (17-83%) and systemic weaknesses in critical areas like model weight security

2. Evaluation Time Compression Crisis: Safety testing compressed from months to days at leading labs, with OpenAI reducing o3 evaluation to less than one week despite advanced capabilities

3. Open-Source Convergence Acceleration: DeepSeek R1's January 2025 release achieved performance parity at 1/27th the cost, fundamentally altering competitive dynamics and safety oversight challenges

4. Safety Team Retention Crisis: 25+ senior safety researchers departed OpenAI in 2024, including entire Superalignment team dissolution, indicating systematic cultural or resource allocation issues

5. Transparency Deterioration: Major models released without promised safety documentation (Google Gemini 2.5 Pro, OpenAI GPT-4.1), violating government commitments

Systemic Concerns

Competitive Pressure Override: Evidence suggests commercial competition is systematically overriding safety considerations across multiple metrics simultaneously.

Voluntary Framework Inadequacy: Current self-regulatory approaches appear insufficient for the scale and pace of capability development.

Information Asymmetry: Companies control both risk evaluation design and disclosure, creating inherent conflicts of interest.

Positive Developments

• First confirmed RSP threshold crossing (Anthropic Claude Opus 4 ASL-3) demonstrates policy operationalization
• Claude 4.5 Sonnet achieved 98.7% safety score with <5% harmful compliance rate
• New York RAISE Act and federal AI Whistleblower Protection Act signal regulatory evolution
• Industry safety team growth (1,100 FTEs vs. 400 in 2022) shows resource commitment expansion