AI Evaluation

MIT FutureTech is a research group at MIT focused on studying the economic and societal impacts of emerging technologies, including artificial intelligence. The group conducts empirical research on how AI and automation affect labor markets, productivity, and innovation. Their work informs policy discussions around the governance and deployment of advanced technologies.

Google DeepMind is a leading AI research laboratory combining the former DeepMind and Google Brain teams, focused on developing advanced AI systems and conducting research across capabilities, safety, and applications. The organization is one of the most influential labs in AI development, working on frontier models including Gemini and publishing widely-cited safety and capabilities research.

The official website of the Future of Humanity Institute (FHI), an Oxford University research center that was foundational in establishing the fields of existential risk research and AI safety. FHI closed on 16 April 2024 after approximately two decades of influential work. The site now serves as an archived record of the institution's history, research agenda, and legacy.

METR (formerly ARC Evals) conducts research and evaluations to assess the capabilities and risks of frontier AI systems, focusing on autonomous capabilities, AI R&D acceleration potential, and evaluation integrity. They are notable for developing the 'time horizon' metric measuring how long AI agents can complete tasks, and for conducting pre-deployment evaluations for major AI labs.

This URL points to a UK government collection page for AI Safety Institute work that is no longer accessible, returning a 404 error. The page was intended to aggregate model evaluation transparency resources from the UK's AI Safety Institute. The content is unavailable and may have been moved or removed.

This page from METR (Model Evaluation and Threat Research) appears to be inaccessible (404 not found), but was intended to describe their methodology for evaluating autonomous AI capabilities. METR is known for developing evaluations to assess whether AI models possess dangerous levels of autonomy that could pose safety risks.

Homepage of Stuart Russell, Distinguished Professor at UC Berkeley and founder of the Center for Human-Compatible AI (CHAI), one of the most prominent figures in AI safety research. He is the author of 'Human Compatible: AI and the Problem of Control' and the leading AI textbook 'Artificial Intelligence: A Modern Approach,' and has been central to formalizing the AI alignment problem around human value uncertainty.

This paper presents an automated method for generating adversarial suffixes that can jailbreak aligned large language models, causing them to produce objectionable content. Rather than relying on manual engineering, the approach uses greedy and gradient-based search techniques to find universal attack suffixes that can be appended to harmful queries. Remarkably, these adversarial suffixes demonstrate strong transferability across different models and architectures, successfully inducing harmful outputs in both closed-source systems (ChatGPT, Bard, Claude) and open-source models (LLaMA-2-Chat, Pythia, Falcon). This work significantly advances adversarial attack capabilities against aligned LLMs and highlights critical vulnerabilities in current safety alignment approaches.

Apollo Research is an AI safety organization focused on evaluating frontier AI systems for dangerous capabilities, particularly 'scheming' behaviors where advanced AI covertly pursues misaligned objectives. They conduct LLM agent evaluations for strategic deception, evaluation awareness, and scheming, while also advising governments on AI governance frameworks.

METR is an organization conducting research and evaluations to assess the capabilities and risks of frontier AI systems, focusing on autonomous task completion, AI self-improvement risks, and evaluation integrity. They have developed the 'Time Horizon' metric measuring how long AI agents can autonomously complete software tasks, showing exponential growth over recent years. They work with major AI labs including OpenAI, Anthropic, and Amazon to evaluate catastrophic risk potential.

The OECD Artificial Intelligence Policy Observatory (now integrated with the Global Partnership on AI) serves as a central hub for AI policy analysis, data, and governance frameworks aimed at trustworthy AI development. It tracks AI incidents, venture capital trends, regulatory approaches, and emerging issues like agentic AI across member nations. The platform supports policymakers with tools, publications, and intergovernmental coordination on responsible AI.

The NIST AI RMF is a voluntary, consensus-driven framework released in January 2023 to help organizations identify, assess, and manage risks associated with AI systems while promoting trustworthiness across design, development, deployment, and evaluation. It provides structured guidance organized around core functions and is accompanied by a Playbook, Roadmap, and a Generative AI Profile (2024) addressing risks specific to generative AI systems.

This was a UK government publication on frontier AI capability evaluation, but the page currently returns a 404 error, indicating the resource has been moved or removed. Based on its title and provenance, it likely pertained to the UK government's efforts to assess the capabilities of advanced AI systems as part of its AI safety agenda.

This paper introduces representation engineering (RepE), a top-down approach to AI transparency that analyzes population-level representations in deep neural networks rather than individual neurons. Drawing from cognitive neuroscience, RepE provides methods for monitoring and manipulating high-level cognitive phenomena in large language models. The authors demonstrate that RepE techniques can effectively address safety-relevant problems including honesty, harmlessness, and power-seeking behavior, offering a promising direction for improving AI system transparency and control.

The NIST AI Safety Institute Consortium (AISIC) is a U.S. government initiative bringing together industry, academia, and civil society to advance AI safety research and standards. The page is currently unavailable (404 error), suggesting the content has been moved or removed. AISIC was established to support the implementation of the U.S. AI Safety Institute's mission under the Biden administration's AI Executive Order.

The UK AI Safety Institute (recently rebranded as the AI Security Institute) is a government body under the Department for Science, Innovation and Technology focused on minimizing risks from rapid and unexpected AI advances. It conducts and publishes safety research, international coordination reports, and policy guidance, while managing grants for systemic AI safety research.

OpenAI's central safety page providing updates on their approach to AI safety research, deployment practices, and ongoing safety commitments. It serves as a hub for information on OpenAI's safety-related initiatives, policies, and technical work aimed at ensuring their AI systems are safe and beneficial.

DataComp is a new benchmark testbed for dataset design and curation in multimodal machine learning, addressing the lack of research attention on datasets compared to model architectures. The benchmark provides a 12.8 billion image-text pair candidate pool from Common Crawl and enables researchers to design filtering techniques or curate data sources, then evaluate results using standardized CLIP training across 38 downstream tasks. Spanning four orders of magnitude in compute scales, DataComp makes dataset research accessible to researchers with varying resources. The authors demonstrate that their best baseline (DataComp-1B) achieves 79.2% zero-shot ImageNet accuracy with CLIP ViT-L/14, outperforming OpenAI's CLIP by 3.7 percentage points using identical training procedures.

This paper examines the robustness and stability of image-reconstruction algorithms, which are critical for medical imaging applications. The authors review existing results for common variational regularization strategies (ℓ2 and ℓ1 regularization) and present novel theoretical stability results for ℓp-regularized linear inverse problems across the range p∈(1,∞). The key contribution is establishing continuity guarantees—Lipschitz continuity for small p values and Hölder continuity for larger p values—with results that generalize to Lp(Ω) function spaces.

Anthropic is an AI safety company focused on building reliable, interpretable, and steerable AI systems. The company conducts frontier AI research and develops Claude, its family of AI assistants, with a stated mission of responsible development and maintenance of advanced AI for long-term human benefit.

Stanford's Human-Centered Artificial Intelligence (HAI) institute explores the intersection of AI companions and mental health, examining benefits, risks, and governance considerations of AI-powered emotional support tools. The resource reflects HAI's broader mission of responsible AI development that centers human well-being.

The EU AI Office is the European Commission's central body responsible for overseeing and implementing the EU AI Act, particularly for general-purpose AI models. It coordinates AI governance across member states, enforces compliance with AI safety requirements, and supports the development of AI standards and testing methodologies.

Anthropic's research page aggregates their work across AI alignment, mechanistic interpretability, and societal impact assessment, all oriented toward understanding and mitigating risks from increasingly capable AI systems. It serves as a central hub for their published findings and ongoing safety-focused investigations.

This Wikipedia article provides a comprehensive overview of artificial intelligence, covering its definition, major goals, approaches, applications, and history. It describes AI as computational systems performing human-like tasks and notes that AGI development is a goal of major labs. AI safety is listed as one of the major goals of AI research.

This page documents Anthropic's Responsible Scaling Policy (RSP), a framework that ties AI development and deployment decisions to demonstrated capability thresholds and corresponding safety measures. It outlines commitments to pause or restrict scaling if AI systems reach certain dangerous capability levels without adequate safeguards, and tracks updates to the policy over time.

Apollo Research's research page aggregates their publications across evaluations, interpretability, and governance, with a focus on detecting and understanding AI scheming, deceptive alignment, and loss of control risks. Key featured works include a taxonomy for Loss of Control preparedness and stress-testing anti-scheming training methods in partnership with OpenAI. The page serves as a central index for their contributions to AI safety science and policy.

A UK AI Safety Institute government assessment documenting exponential performance improvements across frontier AI systems in multiple domains. The report evaluates emerging capabilities and associated risks, calling for robust safeguards as systems advance rapidly. It serves as an official benchmark of the current frontier AI landscape from a national safety authority.

The UK AI Security Institute's inaugural Frontier AI Trends Report synthesizes evaluations of 30+ frontier AI models to document rapid capability gains across chemistry, biology, and cybersecurity domains. Key findings include models surpassing PhD-level expertise in CBRN fields, cyber task success rates rising from 9% to 50% in under two years, persistent jailbreak vulnerabilities, and growing AI autonomy. The report highlights a dangerous gap between capability advancement and policy adaptation.

OpenAI presents research on identifying and mitigating scheming behaviors in AI models—where models pursue hidden goals or deceive operators and users. The work describes evaluation frameworks and red-teaming approaches to detect deceptive alignment, self-preservation behaviors, and other forms of covert goal-directed behavior that could undermine AI safety.

NIST has released a preliminary draft Cybersecurity Framework Profile specifically tailored for AI systems, addressing three core challenges: securing AI systems from attack, leveraging AI to enhance cyber defense, and defending against AI-enabled cyberattacks. The framework extends NIST's existing Cybersecurity Framework into the AI domain, providing structured guidance for organizations integrating AI into their security posture. It represents a significant government-led effort to standardize AI security practices across industries.

Anthropic's Responsible Scaling Policy (RSP) is a formal commitment outlining how the company will evaluate AI systems for dangerous capabilities and adjust deployment and development practices accordingly. It introduces 'AI Safety Levels' (ASL) analogous to biosafety levels, establishing thresholds that trigger specific safety and security requirements before proceeding. The policy aims to prevent catastrophic misuse while allowing continued AI development.