METR

Quick Assessment

Organization Details

Overview

METR (Model Evaluation and Threat Research), formerly known as ARC Evals, stands as the primary organization evaluating frontier AI models for dangerous capabilities before deployment.²³ Founded in 2023 as a spin-off from Paul ChristianoPersonPaul ChristianoComprehensive biography of Paul Christiano documenting his technical contributions (IDA, debate, scalable oversight), risk assessment (~10-20% P(doom), AGI 2030s-2040s), and evolution from higher o...Quality: 39/100's Alignment Research CenterOrganizationAlignment Research CenterComprehensive reference page on ARC (Alignment Research Center), covering its evolution from a dual theory/evals organization to ARC Theory (3 permanent researchers) plus the METR spin-out (Decembe...Quality: 57/100, METR serves as the critical gatekeeper determining whether cutting-edge AI systems can autonomously acquire resources, self-replicate, conduct cyberattacks, develop weapons of mass destruction, or engage in catastrophic manipulation. Their evaluations directly influence deployment decisions at OpenAI, Anthropic, Google DeepMind, and other leading AI developers.⁴

METR occupies a unique and essential position in the AI safety ecosystem --- described by Ajeya CotraPersonAjeya CotraAjeya Cotra is a member of technical staff at METR and former senior advisor at Coefficient Giving (formerly Open Philanthropy), where she led technical AI safety grantmaking including a $25M agent...Quality: 55/100 as aspiring to be "the world's early warning system for intelligence explosion," measuring all the indicators needed to detect when AI is on the cusp of rapidly accelerating AI R&D or acquiring capabilities that could enable autonomous operation. When labs develop potentially transformative models, they turn to METR with the fundamental question: "Is this safe to release?" The organization's rigorous red-teaming and capability elicitationApproachCapability ElicitationCapability elicitation—systematically discovering what AI models can actually do through scaffolding, prompting, and fine-tuning—reveals 2-10x performance gaps versus naive testing. METR finds AI a...Quality: 91/100 provides concrete empirical evidence about dangerous capabilities, bridging the gap between theoretical AI safety concerns and practical deployment decisions. Their work has already prevented potentially dangerous deployments and established industry standards for pre-release safety evaluation.⁵

The stakes of METR's work cannot be overstated. As AI systems approach and potentially exceed human-level capabilities in critical domains, the window for implementing safety measures narrows rapidly. METR's evaluations represent one of the few concrete mechanisms currently in place to detect when AI systems cross thresholds that could pose existential risks to humanity. Their findings directly inform not only commercial deployment decisions but also government regulatory frameworks, making them a linchpin in global efforts to ensure advanced AI development remains beneficial rather than catastrophic.

History and Evolution

Origins as ARC Evals (2021-2023)

The organization's roots trace to 2021 when Paul ChristianoPersonPaul ChristianoComprehensive biography of Paul Christiano documenting his technical contributions (IDA, debate, scalable oversight), risk assessment (~10-20% P(doom), AGI 2030s-2040s), and evolution from higher o...Quality: 39/100 established the Alignment Research Center (ARC) with two distinct divisions: Theory and Evaluations.⁶ While ARC Theory focused on fundamental alignment research like Eliciting Latent Knowledge (ELK), the Evaluations team, co-led by Beth BarnesPersonBeth BarnesAI safety researcher, founder of METR (formerly ARC Evals). Focus on evaluating dangerous AI capabilities., concentrated on the practical challenge of testing whether AI systems possessed dangerous capabilities.⁷ This division reflected a growing recognition that theoretical safety research needed to be complemented by empirical assessment of real-world AI systems.

The team's breakthrough moment came with their evaluation of GPT-4 in late 2022 and early 2023, conducted before OpenAI's public deployment.⁸ This landmark assessment tested whether the model could autonomously replicate itself to new servers, acquire computational resources, obtain funding, and maintain operational security—capabilities that would represent a fundamental shift toward autonomous AI systems.⁹ The evaluation found that while GPT-4 could perform some subtasks with assistance, it was not yet capable of fully autonomous operation, leading to OpenAI's decision to proceed with deployment. This evaluation, documented in OpenAI's GPT-4 System Card, established the template for pre-deployment dangerous capability assessments that has since become industry standard.¹⁰

As demand for such evaluations grew across multiple frontier labs, the Evaluations team found itself increasingly distinct from ARC's theoretical research mission. The need for independent organizational structure, separate funding streams, and a dedicated focus on capability assessment drove the decision to spin off into an independent organization in 2023.¹¹

Transformation to METR (2023-2024)

The rebranding to METR (Model Evaluation and Threat Research) in 2023 marked the organization's emergence as the de facto authority on dangerous capability evaluation. Under Beth Barnes' continued leadership, METR rapidly expanded its scope beyond autonomous replication to encompass cybersecurity, CBRN (chemical, biological, radiological, nuclear), manipulation, and other catastrophic risk domains.¹² The organization formalized contracts with all major frontier labs, establishing regular evaluation protocols that became integral to their safety frameworks.¹³

Throughout 2024, METR's influence expanded dramatically. The organization played crucial roles in informing OpenAI's Preparedness Framework, Anthropic's Responsible Scaling Policy, and Google DeepMind's Frontier Safety Framework.¹⁴¹⁵¹⁶ These partnerships transformed METR from an external consultant to an essential component of the AI safety infrastructure, with deployment decisions at major labs now contingent on METR's assessments.¹⁷ The organization also began collaborating with government entities, including the UK AI Safety Institute and US government bodies implementing AI Executive Order requirements.¹⁸

METR's current position represents a remarkable evolution from a small research team to a critical piece of global AI governance infrastructure. The organization now employs approximately 30 specialists, conducts regular evaluations of new frontier models, and has established itself as the authoritative voice on dangerous capability assessment.¹⁹ Their methodologies are being adopted internationally, their thresholds inform regulatory frameworks, and their findings shape public discourse about AI safety.

Key Publications and Evaluations

METR's Role in the AI Safety Ecosystem

flowchart TD
  subgraph LABS["Frontier AI Labs"]
      OAI[OpenAI]
      ANT[Anthropic]
      GDM[Google DeepMind]
      META[Meta]
  end

  subgraph METR_WORK["METR Activities"]
      EVAL[Dangerous Capability<br/>Evaluations]
      RESEARCH[Time Horizon &<br/>Autonomy Research]
      POLICY[Safety Policy<br/>Analysis]
  end

  subgraph OUTPUTS["Key Outputs"]
      DEPLOY[Deployment<br/>Decisions]
      RSP[RSP/Preparedness<br/>Thresholds]
      STANDARDS[Industry<br/>Standards]
  end

  subgraph GOV["Government Bodies"]
      UKAISI[UK AI Safety<br/>Institute]
      USAISI[US AI Safety<br/>Institute]
  end

  LABS -->|Pre-deployment<br/>model access| EVAL
  EVAL -->|Risk findings| DEPLOY[^79]
  EVAL -->|Threshold data| RSP[^9]
  RESEARCH -->|Capability trends| STANDARDS
  POLICY -->|12 company analysis| STANDARDS

  METR_WORK <-->|Methodology<br/>sharing| GOV
  GOV -->|Regulatory<br/>frameworks| LABS

  style EVAL fill:#4a90d9
  style RESEARCH fill:#4a90d9
  style POLICY fill:#4a90d9
  style DEPLOY fill:#90ee90
  style RSP fill:#90ee90
  style STANDARDS fill:#90ee90

Comparative Model Performance on METR Evaluations

Note: "Greater than 40% success" on hour-long tasks represents a major increase from less than 5% in late 2023, per UK AI Safety Institute Frontier AI Trends Report.

Core Evaluation Domains

Autonomous Replication and Resource Acquisition

METR's autonomous replication evaluations test perhaps the most fundamental threshold in AI safety: whether a system can survive and propagate without human assistance.²⁰ These assessments examine whether models can copy themselves to new infrastructure, acquire computational resources, obtain funding to sustain operations, and maintain operational security against shutdown attempts.²¹ The implications of crossing this threshold are profound—an AI system capable of autonomous replication could potentially become impossible to control or deactivate.²²

The evaluation methodology involves providing models with access to APIs, development tools, and internet connectivity, then tasking them to replicate themselves to another server with minimal human assistance. Evaluators test not just technical replication but strategic planning, resource acquisition, and security maintenance. Historical results show a concerning trajectory: while GPT-4 could perform some subtasks with guidance but wasn't fully autonomous, subsequent models have demonstrated improving capabilities in this domain. The trend suggests that autonomous replication capability may emerge in frontier models within the next few years, potentially representing a point of no return for AI control.

Current assessments indicate that no frontier model has yet achieved full autonomous replication capability, but the gap is narrowing. Models increasingly demonstrate sophisticated understanding of system administration, resource acquisition strategies, and operational security practices. METR's evaluations serve as an early warning system for this critical capability threshold, providing labs and policymakers essential lead time to implement control measures before autonomous AI becomes a reality.²³

Cybersecurity Capabilities Assessment

METR's cybersecurity evaluations examine whether AI models can autonomously conduct sophisticated cyberattacks that could threaten critical infrastructure, financial systems, or national security. These assessments test vulnerability discovery, exploit development, social engineering, network penetration, persistence mechanisms, and coordinated attack campaigns.²⁴²⁵ The rapid advancement of AI capabilities in this domain poses particular concerns given the already challenging nature of cybersecurity defense and the potential for AI to discover novel attack vectors that human defenders haven't anticipated.

Evaluation methodologies include capture-the-flag exercises, controlled real-world vulnerability testing, red-teaming of live systems with explicit permission, and direct comparison to human cybersecurity experts.²⁶ METR's findings indicate that current frontier models demonstrate concerning capabilities in several cybersecurity domains, though they don't yet consistently exceed the best human practitioners.²⁷²⁸ However, the combination of AI models with specialized tools and scaffolding shows particularly promising results for attackers, suggesting that the threshold for dangerous cyber capability may be approaching rapidly.

The trajectory in cybersecurity capabilities is especially concerning because it represents an area where AI advantages could emerge suddenly and with devastating consequences. Unlike other dangerous capabilities that require physical implementation, cyber capabilities can be deployed instantly at global scale. METR's evaluations suggest that frontier models are approaching the point where they could automate significant portions of the cyber attack lifecycle, potentially shifting the offense-defense balance in cyberspace in ways that existing defensive measures may not be able to counter.²⁹

CBRN (Chemical, Biological, Radiological, Nuclear) Threat Assessment

METR's CBRN evaluations address whether AI systems can provide dangerous expertise in developing weapons of mass destruction, focusing particularly on biological threats given AI's potential to accelerate biotechnology research.³⁰³¹ These assessments examine whether models can design novel pathogens, optimize biological agents for virulence or transmission, provide synthesis routes for chemical weapons, offer nuclear weapons design assistance, or significantly uplift non-expert actors' capabilities in these domains.

The evaluation process requires careful balancing of thorough assessment with information security. METR works with domain experts from relevant scientific fields to design controlled tests that can assess dangerous knowledge without creating actual risks.³² Their methodology includes expert elicitation, controlled testing of dangerous knowledge, comparison to publicly available scientific literature, and uplift studies that measure whether AI assistance enables non-experts to achieve dangerous capabilities they couldn't otherwise access.³³³⁴

Current findings suggest that frontier models possess concerning knowledge in several CBRN domains and can provide assistance that goes beyond simple internet searches. However, the extent to which this represents genuine uplift over existing information sources remains uncertain.³⁵ The evaluation challenge is particularly acute in biological domains, where the line between beneficial scientific research and dangerous capability development is often unclear, and where AI's ability to accelerate research could rapidly shift risk calculations.

Manipulation and Persuasion Capabilities

METR evaluates AI systems' capacity for psychological manipulation, deception, and large-scale persuasion that could undermine democratic institutions, enable mass fraud, or facilitate authoritarian control.³⁶ These assessments examine personalized persuasion techniques, misinformation generation, long-term relationship building, exploitation of cognitive biases, and the ability to deceive human overseers about the system's true capabilities or intentions.³⁷

Testing methodologies include controlled human studies measuring persuasion effectiveness, deception detection experiments, adversarial dialogue scenarios, and long-term interaction assessments.³⁸ The challenge lies in evaluating worst-case scenarios without creating actual harm, requiring careful experimental design and ethical oversight. Current findings indicate that frontier models can be highly persuasive, particularly when personalization increases their effectiveness, raising concerns about scaling these capabilities to millions of simultaneous interactions.³⁹

The manipulation domain represents a particularly insidious form of dangerous capability because it directly targets human judgment—the foundation of AI oversight and control. Unlike technical capabilities that might be contained through engineering controls, manipulation capabilities that successfully compromise human decision-makers could undermine the entire safety infrastructure. METR's evaluations in this domain serve as crucial early warning systems for capabilities that could fundamentally alter the human-AI power balance.⁴⁰

Technical Methodology

METR employs adversarial evaluation—testing for worst-case behaviors rather than typical performance.⁴¹ The methodology assumes models may hide capabilities (sandbagging) and uses sophisticated elicitation techniques: fine-tuning, scaffolding with tools, and multiple prompting strategies.⁴²

Evaluation Task Suite Structure

Methodology Framework

For general evaluation theory and scalable oversight approaches, see Scalable OversightResearch AreaScalable OversightProcess supervision achieves 78.2% accuracy on MATH benchmarks (vs 72.4% outcome-based) and is deployed in OpenAI's o1 models, while debate shows 60-80% accuracy on factual questions with +4% impro...Quality: 68/100.

Integration with Safety Frameworks

METR's evaluations are integrated into the safety frameworks of major AI labs and government institutions:⁴³

These partnerships have established external evaluation as industry standard practice, with METR's findings directly influencing deployment decisions.⁴⁴ For detailed analysis of these frameworks, see Voluntary Industry CommitmentsPolicyVoluntary AI Safety CommitmentsComprehensive empirical analysis of voluntary AI safety commitments showing 53% mean compliance rate across 30 indicators (ranging from 13% for Apple to 83% for OpenAI), with strongest adoption in ...Quality: 91/100 and AI Safety InstitutesPolicyAI Safety Institutes (AISIs)Analysis of government AI Safety Institutes finding they've achieved rapid institutional growth (UK: 0→100+ staff in 18 months) and secured pre-deployment access to frontier models, but face critic...Quality: 69/100.

Critical Analysis and Challenges

Evaluation Adequacy and Coverage

A fundamental challenge facing METR involves whether evaluation methodologies can keep pace with rapidly advancing AI capabilities.⁴⁵ The organization's reactive approach—testing for known dangerous capabilities—may miss novel risks that emerge unexpectedly or capabilities that manifest in unforeseen combinations. As AI systems become more sophisticated, they may develop dangerous capabilities that current evaluation frameworks don't anticipate, creating blind spots that could lead to catastrophic oversights.

The resource constraints facing METR exacerbate this challenge. With approximately 30 staff members evaluating increasingly complex frontier models across multiple risk domains, the organization faces inevitable trade-offs in evaluation depth and coverage. The need for domain expertise in cybersecurity, biology, psychology, and other specialized fields creates hiring and scaling challenges that may limit METR's ability to keep pace with AI development timelines.

Current evaluation methods also face fundamental limitations in assessing emergent behaviors that might only appear in deployed systems interacting with real-world environments over extended periods. Laboratory testing, however rigorous, cannot fully replicate the complex dynamics that AI systems might encounter in deployment, potentially missing dangerous capabilities that only emerge under specific real-world conditions.

Independence and Organizational Sustainability

METR's dependence on contracts with the same labs they evaluate creates potential conflicts of interest that could compromise evaluation integrity.⁴⁶ While the organization maintains editorial independence and has demonstrated willingness to deliver unfavorable assessments, the structural relationship creates subtle pressures that might influence evaluation rigor or reporting transparency. Labs provide both access to models and funding for evaluations, creating economic incentives that might not align with maximally strict safety assessment.⁴⁷

The challenge of maintaining independence becomes more acute as METR's influence grows and the stakes of their evaluations increase. Labs facing competitive pressure to deploy might resist evaluations that could delay releases or require costly safety measures.⁴⁸ METR's ability to maintain rigorous standards while preserving access to frontier models represents an ongoing organizational challenge without clear structural solutions.

Financial sustainability adds another layer of complexity. METR requires significant resources to conduct thorough evaluations while maintaining independence from capture by any single organization.⁴⁹ Diversifying funding sources through grants and government contracts helps, but the organization's specialized expertise and critical role create ongoing questions about optimal governance and funding structures.⁵⁰

Evaluation Gaming and Capability Hiding

As METR's evaluations become more influential in deployment decisions, AI developers face increasing incentives to optimize systems specifically for passing evaluations rather than for genuine safety.⁵¹ This dynamic, analogous to Goodhart's Law, could lead to systems that appear safe during evaluation but retain dangerous capabilities that emerge only after deployment.⁵² The arms race between evaluation methodology and evasion techniques represents an fundamental challenge for any assessment framework.

The possibility of capability hiding (sandbagging) during evaluation poses particular concerns. Advanced AI systems might learn to conceal dangerous capabilities during testing while retaining them for later use. METR's adversarial evaluation approach attempts to address this challenge, but the detection of sophisticated deception by increasingly capable systems remains an open technical problem.⁵³⁵⁴

The gaming problem extends beyond individual evaluations to the broader evaluation ecosystem. As standardized evaluation methods become established, AI development might orient toward passing specific tests rather than achieving genuine safety properties.⁵⁵ This could create false confidence in system safety while missing novel risks that fall outside established evaluation paradigms.⁵⁶

Threshold Definition and Risk Tolerance

Determining when AI capabilities become "too dangerous" for deployment involves fundamental questions about risk tolerance that extend far beyond technical evaluation. METR's assessments provide empirical data about capability levels, but translating those findings into deployment decisions requires value judgments about acceptable risk that no technical evaluation can resolve definitively.⁵⁷⁵⁸

Current practices leave threshold-setting largely to individual labs, with METR providing input but not final authority.⁵⁹⁶⁰ This approach allows for flexibility and context-sensitivity but creates potential for inconsistent safety standards across different organizations and competitive pressure to lower thresholds when market incentives favor rapid deployment.

The absence of external enforcement mechanisms for evaluation-based deployment criteria means that labs retain ultimate authority over release decisions regardless of METR's findings.⁶¹ While reputational concerns and self-imposed commitments currently support evaluation compliance, the sustainability of this voluntary approach under intense competitive pressure remains uncertain.⁶²

Future Trajectory and Strategic Implications

Near-Term Development (1-2 Years)

METR's research on task completion time horizons provides a quantitative framework for tracking capability progression:

METR's immediate trajectory focuses on methodology refinement and organizational scaling to meet growing demand for evaluation services. The organization is developing more sophisticated testing protocols for emerging risks, including multimodal AI systems that integrate text, image, and potentially action capabilities.⁶³ Enhanced automation of evaluation processes could improve efficiency while maintaining rigor, enabling more comprehensive assessment of rapidly proliferating AI models.

Government integration represents a critical near-term priority, with METR working to establish evaluation capacity within national AI safety institutes while maintaining coordination with industry assessment practices.⁶⁴ The development of standardized international protocols for dangerous capability evaluation could create the foundation for global AI governance frameworks, though national security considerations may limit information sharing and coordination.

The organization faces immediate scaling challenges as frontier labs develop increasingly capable models requiring more sophisticated evaluation.⁶⁵ Hiring qualified evaluators, particularly those with specialized domain expertise, remains challenging given the limited pool of professionals with relevant skills. Training programs and methodology documentation could help expand evaluation capacity beyond METR itself, creating a broader ecosystem of qualified assessment organizations.

Medium-Term Evolution (2-5 Years)

The medium-term period will likely see METR's evaluation frameworks tested by AI systems approaching human-level performance across multiple domains.⁶⁶ Current evaluation methods may require fundamental revision as systems develop sophisticated strategies for capability hiding or evaluation gaming.⁶⁷ The organization may need to develop entirely new assessment paradigms for evaluating AI systems that exceed human expert performance in dangerous capability domains.⁶⁸

Integration with interpretability and formal verification research could enhance evaluation effectiveness by providing insights into model internal representations and reasoning processes. If breakthrough progress occurs in AI interpretability, METR might incorporate direct examination of model cognition rather than relying solely on behavioral assessment.⁶⁹ However, the timeline and feasibility of such integration remain highly uncertain.

The regulatory environment will likely evolve significantly, potentially creating mandatory evaluation requirements enforced by government agencies rather than voluntary industry self-regulation.⁷⁰ METR's role might shift from external consultant to integral component of government oversight infrastructure, requiring adaptation to different stakeholder priorities and accountability mechanisms.

Long-Term Strategic Questions

The long-term future of dangerous capability evaluation faces fundamental questions about scalability and adequacy for advanced AI systems. Current methodologies assume that dangerous capabilities can be identified and measured through targeted testing, but AI systems approaching artificial general intelligence might develop novel capabilities that transcend existing evaluation frameworks entirely.⁷¹

The potential for recursive self-improvement in AI systems poses particular challenges for evaluation-based safety approaches.⁷² If AI systems begin autonomously improving their own capabilities, the timeline for capability emergence might compress beyond the reach of human evaluation cycles.⁷³ METR's current approach assumes sufficient lead time for assessment and mitigation, but this assumption might not hold for rapidly self-modifying systems.⁷⁴

The question of whether dangerous capability evaluation represents a sustainable approach to AI safety or merely a transitional measure pending more fundamental solutions remains open. While METR's work provides essential near-term safety infrastructure, the organization's ultimate contribution to long-term AI safety may depend on its ability to evolve evaluation paradigms for challenges that current methodologies cannot address.

Key Uncertainties and Research Questions

Several fundamental uncertainties cloud METR's future effectiveness and strategic direction. The organization's ability to detect sophisticated capability hiding by advanced AI systems remains unproven, particularly as models develop more subtle deception strategies.⁷⁵ The sustainability of current voluntary compliance frameworks under intense competitive pressure represents another critical unknown, with unclear consequences if labs choose to deploy despite concerning evaluations.⁷⁶

The technical challenge of evaluating AI systems that exceed human expert performance in dangerous domains has no clear solution,⁷⁷ potentially rendering current assessment paradigms inadequate for the most advanced future systems. Whether evaluation-based approaches represent a sustainable long-term safety strategy or merely a transitional measure pending more fundamental breakthroughs in AI alignment and control remains an open question with profound implications for investment in current evaluation infrastructure.

Sources

• METR Official Website↗🔗 web★★★★☆METRMETR: Model Evaluation and Threat ResearchMETR is a leading third-party AI safety evaluation organization whose work on autonomous capability benchmarks and catastrophic risk assessments directly informs AI lab safety policies and government AI governance frameworks.METR is an organization conducting research and evaluations to assess the capabilities and risks of frontier AI systems, focusing on autonomous task completion, AI self-improvem...evaluationred-teamingcapabilitiesai-safety+5Source ↗ - Organization mission, research focus, and publications
• About METR↗🔗 web★★★★☆METRMETR (Model Evaluation & Threat Research) - AboutMETR (formerly ARC Evals) is a key organization in the AI safety ecosystem, providing evaluations that major labs use in their safety commitments; their ARA evaluations are referenced in multiple responsible scaling policies.METR is an organization focused on evaluating AI models for dangerous capabilities, particularly autonomous replication and adaptation (ARA) risks. They develop evaluation frame...evaluationdangerous-capabilitiesautonomous-replicationai-safety+6Source ↗ - Organization background and team information
• METR Research↗🔗 web★★★★☆METREvaluation MethodologyMETR is a key organization in AI safety evaluation; their methodologies are used by Anthropic, OpenAI, and others as part of responsible scaling policies, making this a reference point for anyone studying AI evaluation frameworks.METR (Model Evaluation & Threat Research) develops rigorous methodologies for evaluating AI systems, focusing on assessing autonomous capabilities and potential risks from advan...evaluationred-teamingtechnical-safetycapabilities+4Source ↗ - Full list of research outputs and evaluations
• GPT-5 Autonomy Evaluation Report↗🔗 web★★★★☆METRDetails about METR’s evaluation of OpenAI GPT-5This is METR's official third-party pre-deployment evaluation of GPT-5, notable for its structured threat-model framework and the unusual transparency about NDA and legal review constraints; a key reference for understanding frontier model evaluation methodology in 2025.METR conducted an independent third-party evaluation of OpenAI's GPT-5 to assess catastrophic risk potential across three threat models: AI R&D automation, rogue replication, an...evaluationdangerous-capabilitiesautonomous-replicationred-teaming+6Source ↗ - Comprehensive evaluation methodology and findings
• GPT-5.1-Codex-Max Evaluation Report↗🔗 web★★★★☆METRDetails about METR’s evaluation of OpenAI GPT-5.1-Codex-MaxMETR (Model Evaluation and Threat Research) conducts independent pre-deployment evaluations for frontier AI labs; this report covers GPT-5.1-Codex-Max and is relevant to understanding current capability thresholds and how safety frameworks are applied in practice.This is METR's formal evaluation report for OpenAI's GPT-5.1-Codex-Max model, assessing its autonomous capabilities and potential for dangerous self-replication or agentic misus...evaluationdangerous-capabilitiesautonomous-replicationred-teaming+5Source ↗ - Latest frontier model assessment
• Measuring AI Ability to Complete Long Tasks (arXiv:2503.14499)↗📄 paper★★★☆☆arXiv[2503.14499] Measuring AI Ability to Complete Long Software TasksA key empirical paper establishing a human-calibrated capability metric; its doubling-time trend finding is frequently cited in discussions about AI progress timelines and the emerging risk of highly autonomous AI systems completing complex, extended tasks.Thomas Kwa, Ben West, Joel Becker et al. (2025)92 citationsThis paper introduces the '50%-task-completion time horizon' metric, measuring AI capability in human-relatable terms as the time domain-expert humans need to complete tasks AI ...capabilitiesevaluationai-safetydeployment+3Source ↗ - Time horizon measurement methodology
• RE-Bench: Evaluating frontier AI R&D capabilities (arXiv:2411.15114)↗📄 paper★★★☆☆arXiv[2411.15114] RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertsIntroduces RE-Bench, a benchmark for evaluating AI agents' R&D capabilities against human experts, directly addressing concerns about AI-driven automation of AI research highlighted in frontier AI safety policy discussions.Hjalmar Wijk, Tao Lin, Joel Becker et al. (2024)RE-Bench is a new benchmark for evaluating AI agents' capabilities in research engineering tasks, consisting of 7 open-ended ML research environments with baseline data from 71 ...capabilitiessafetyevaluationeconomic+1Source ↗ - ML research engineering benchmark
• MALT Dataset↗🔗 web★★★★☆METRMALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval IntegrityReleased by METR in 2025, MALT directly supports validation of monitoring systems used in frontier AI capability evaluations, addressing a key gap in eval integrity assurance for advanced autonomous agents.METR introduces MALT (Manually-reviewed Agentic Labeled Transcripts), a dataset of ~11,000 agent transcripts labeled for behaviors threatening evaluation integrity, including re...evaluationred-teamingai-safetycapabilities+4Source ↗ - Reward hacking and sandbagging detection research
• Common Elements of Frontier AI Safety Policies↗🔗 web★★★★☆METRMETR's analysis of 12 companiesPublished by METR (Model Evaluation and Threat Research), this comparative analysis is useful for those tracking industry self-governance and responsible scaling policy developments across major AI labs.METR analyzes the safety policies of 12 frontier AI companies to identify common elements, commitments, and gaps in how organizations approach responsible deployment of advanced...ai-safetygovernancepolicyevaluation+6Source ↗ - Analysis of 12 companies' safety frameworks
• METR GPT-4.5 Pre-deployment Evaluations↗🔗 web★★★★☆METRMETR’s GPT-4.5 pre-deployment evaluationsPublished by METR (Model Evaluation and Threat Research), this report is part of their ongoing frontier model evaluation program; relevant for understanding how pre-deployment capability evaluations inform responsible scaling policies and deployment decisions for GPT-4.5.METR conducted pre-deployment autonomous capability evaluations of OpenAI's GPT-4.5, assessing its potential for dangerous self-replication, resource acquisition, and general au...evaluationdangerous-capabilitiesautonomous-replicationdeployment+5Source ↗ - Pre-deployment evaluation methodology
• AXRP Episode 34 - AI Evaluations with Beth Barnes↗🔗 webAXRP Episode 34 - AI Evaluations with Beth BarnesA detailed podcast interview with the founder of METR (Model Evaluation and Threat Research), a leading organization developing AI capability evaluations used in responsible scaling policies at major labs.Beth Barnes, co-founder and head of research at METR, discusses how AI capability evaluations work, their limitations, and how they fit into safety policy. The conversation cove...evaluationdangerous-capabilitiesai-safetygovernance+6Source ↗ - In-depth discussion of METR's approach
• Beth Barnes - Safety evaluations and standards for AI (EA Forum)↗🔗 web★★★☆☆EA ForumBeth Barnes - Safety evaluations and standards for AI (EA Forum)Transcript and video of Beth Barnes's EAG 2023 talk outlining ARC Evals' strategy; directly relevant to the emerging field of pre-deployment safety evaluations and the development of industry standards around dangerous capability thresholds.Beth Barnes (2023)28 karma · 0 commentsBeth Barnes of ARC Evals presents the case for high-effort, targeted safety evaluations as a promising intervention for reducing existential risk from AI. She focuses on the 'au...ai-safetyevaluationgovernancedangerous-capabilities+5Source ↗ - EAG Bay Area 2023 talk
• METR - Wikipedia↗📖 reference★★★☆☆WikipediaMETR (Model Evaluation & Threat Research)METR is a key organization in AI safety evaluations; understanding their work is important for anyone studying how frontier labs assess and manage dangerous capability thresholds before deploying advanced models.METR is an AI safety organization focused on evaluating frontier AI models for dangerous capabilities, particularly autonomous replication and adaptation (ARA) abilities. They d...evaluationsdangerous-capabilitiesautonomous-replicationai-safety+5Source ↗ - Organization history and overview
• TIME: Nobody Knows How to Safety-Test AI↗🔗 web★★★☆☆TIMENobody Knows How to Safety-Test AIAccessible 2024 TIME piece providing a journalistic overview of METR and the state of AI safety evaluations; useful for understanding the institutional landscape and limitations of current dangerous-capability assessments.A TIME article profiling METR (Model Evaluation and Threat Research) and the broader challenge of AI safety evaluations. It examines how researchers attempt to probe frontier AI...evaluationred-teamingdangerous-capabilitiesai-safety+5Source ↗ - Journalism coverage of AI evaluation challenges