UK AI Safety Institute

Overview

The UK AI Safety Institute (UK AISI) is a government organization established in November 2023 to advance AI safety through research, evaluation, and international coordinationAi Transition Model ParameterInternational CoordinationThis page contains only a React component placeholder with no actual content rendered. Cannot assess importance or quality without substantive text.. Created in the wake of the first AI Safety Summit at Bletchley Park, AISI represents the UK's commitment to being a global leader in AI safety and governance. In February 2025, the organization was renamed the AI Security Institute to reflect a sharper focus on national security threats including cyberattacks, fraud, and AI-enabled bioweapon development.

As one of the first government bodies specifically dedicated to advanced AI safety (alongside the US AISI), the UK AISI plays a pioneering role in translating AI safety research into government policy and practice. The organization conducts technical research on AI risks, evaluates frontier models for dangerous capabilities, develops safety standards, and coordinates international efforts through the International Network of AI Safety InstitutesPolicyAI Safety Institutes (AISIs)Analysis of government AI Safety Institutes finding they've achieved rapid institutional growth (UK: 0→100+ staff in 18 months) and secured pre-deployment access to frontier models, but face critic...Quality: 69/100, which now includes 10+ member countries.

The UK AISI's strategic positioning reflects the UK government's ambition: to be a global hub for AI safety, bridging technical research communities, frontier AI labs, and international policymakers. Chair Ian Hogarth has described AISI as a "startup inside the government" that combines technical rigor with policy influence.

Organization at a Glance

History and Founding

Background: UK AI Safety Summit (November 2023)

Bletchley Park Declaration:

• UK hosted first international AI Safety Summit at Bletchley Park
• 28 countries and EU signed declaration on AI risks
• Recognized catastrophic risks from advanced AI
• Committed to international cooperation on safety
• Political moment crystallizing government attention

Summit outcomes:

• Agreement on establishing AI Safety Institutes
• UK and US announced creation of their institutes
• Commitments from frontier AI labs
• International research collaboration agreements
• Foundation for ongoing coordination

Context:

• ChatGPT moment (late 2022) raised AI awareness
• Growing concern about AI existential risk
• UK positioning for leadership role
• Rishi Sunak government prioritizing AI policy
• Academic and researcher advocacy for government action

Establishment (2023)

Founding: Announced at AI Safety Summit, operationalized late 2023

Organizational placement:

• Part of UK Department for Science, Innovation and Technology (DSIT)
• Government agency with civil service staff
• Independent research capacity
• Policy coordination role
• International engagement mandate

Initial mission:

• Conduct safety research on advanced AI
• Evaluate frontier models for dangerous capabilities
• Develop testing and evaluation methodologies
• Coordinate international AI safety efforts
• Advise UK government on AI risks and governance

Funding: The UK initially invested 100 million GBP in the Frontier AI Taskforce (AISI's predecessor), providing more funding for AI safety than any other country at the time. The Institute's annual budget is approximately 50 million GBP, continuing as an annual amount through the end of the decade, subject to demonstrating continued requirement. This makes the UK AISI an outlier in funding compared to other national AI Safety Institutes globally.

Early Development (2023-2024)

2023 activities: The Institute rapidly hired technical staff, recruiting senior alumni from OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to commercial AGI developer, with detailed analysis of governance crisis, safety researcher exodus (75% of ..., Google DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100, and the University of Oxford. Ian Hogarth was appointed as permanent Chair in June 2023. By the end of 2023, the organization had established its organizational structure, built relationships with AI labs for model access, and begun coordination with the US AISI.

2024 milestones:

Current status (2025): Following the February 2025 renaming to AI Security Institute, the organization now focuses more explicitly on national security threats. Technology Secretary Peter Kyle announced the change at the Munich Security Conference, emphasizing a pivot from concerns about bias and misinformation toward risks such as cyberattacks, fraud, and AI-enabled bioweapon development. The Institute has launched a new criminal misuse team in partnership with the Home Office to investigate AI's role in serious crimes including child exploitation, cybercrime, and financial fraud.

Core Functions and Activities

1. Frontier Model Evaluations

The Institute independently assesses advanced AI systems for dangerous capabilities across domains critical to national security and public safety. In their April 2024 evaluation of publicly available frontier models, AISI found several concerning capabilities:

Methodology: The Institute has built evaluation expertise by learning from and collaborating with organizations like METROrganizationMETRMETR conducts pre-deployment dangerous capability evaluations for frontier AI labs (OpenAI, Anthropic, Google DeepMind), testing autonomous replication, cybersecurity, CBRN, and manipulation capabi...Quality: 66/100, Apollo ResearchOrganizationApollo ResearchApollo Research demonstrated in December 2024 that all six tested frontier models (including o1, Claude 3.5 Sonnet, Gemini 1.5 Pro) engage in scheming behaviors, with o1 maintaining deception in ov...Quality: 58/100, and ARC. Their approach includes both pre-deployment and post-deployment testing, with a focus on red-teaming and adversarial evaluation techniques.

Lab cooperation: AISI has established voluntary agreements with OpenAI, AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding..., and Google DeepMind for pre-release access to models. Labs share safety-relevant information under NDAs, coordinate on safety thresholds, and collaborate on evaluation methodology development.

Inspect AI Framework: In May 2024, AISI open-sourced Inspect AI, a Python framework for building and running reproducible LLM evaluations. The framework has been adopted by major AI labs including Anthropic, DeepMind, and xAI. Inspect AI includes over 100 pre-built evaluations, a web-based visualization tool, and support for tool calling and multi-agent evaluations. Key benchmarks include ChemBench (2,786 chemistry questions), Humanity's Last Exam (3,000 questions across dozens of subjects), and Cybench (40 professional-level cybersecurity challenges).

2. Research Funding Programs

AISI manages substantial external research funding, making it one of the largest funders of AI safety research globally. The Institute's grant programs combine government funding with partnerships from industry and philanthropy.

The Alignment Project: This global fund is supported by an international coalition including the UK AISI, Canadian AI Safety Institute, Schmidt Sciences, Amazon Web Services, Halcyon Futures, Safe AI Fund, UKRI, Anthropic, and ARIA. Recipients can also receive up to 5 million GBP in dedicated AWS cloud computing credits, enabling technical experiments beyond typical academic reach.

Systemic Safety Grants: Launched in partnership with the Engineering and Physical Sciences Research Council (EPSRC) and Innovate UK (part of UKRI), the first phase allocated 4 million GBP to support approximately 20 projects. The program aims to increase societal resilience to widespread AI deployment across critical infrastructure.

3. Technical Safety Research

AISI conducts both in-house and collaborative research across several priority areas:

Technical research: Evaluation methodologies, interpretability and transparency techniques, robustness and reliability testing, alignment approaches, and measurement/benchmarking standards.

Sociotechnical research: AI governanceParameterAI GovernanceThis page contains only component imports with no actual content - it displays dynamically loaded data from an external source that cannot be evaluated. frameworks, risk assessment methodologies, safety standards and best practices, international coordination mechanismsPolicyInternational Coordination MechanismsComprehensive analysis of international AI coordination mechanisms shows growing but limited progress: 11-country AI Safety Institute network with ~$200M budget expanding to include India; Council ...Quality: 91/100, and public engagement strategies.

Key outputs include: The Frontier AI Trends Report (December 2024), which provides an empirical assessment of frontier AI capabilities and their trajectory, and regular evaluation reports shared with policymakers and the public.

4. International Coordination

The UK has positioned itself as a convener for international AI safety efforts, hosting the first AI Safety Summit and co-founding the International Network of AI Safety Institutes.

International Network of AI Safety Institutes: During the AI Seoul Summit in May 2024, international leaders agreed to form this network comprising institutes from the UK, US, Japan, France, Germany, Italy, Singapore, South Korea, Australia, Canada, and the European Union. The first meeting was held in San Francisco in November 2024, with representatives from 10 countries and the European Commission.

US-UK partnership: The UK and US AISIs work as "sister organizations" with close collaboration on methodologies, coordinated evaluations, and joint international engagement. The San Francisco office (opened summer 2024) facilitates this partnership and allows AISI to recruit Bay Area talent and engage directly with major AI labs.

Seoul Statement goals: The international network aims to "accelerate the advancement of the science of AI safety" by promoting complementarity and interoperability between institutes and fostering a common international understanding of AI safety approaches.

5. Standard-Setting and Guidelines

Developing best practices:

Evaluation standards:

• Methodologies for dangerous capability testing
• Thresholds for deployment decisions
• Red-teaming protocols
• Transparency and documentation requirements
• Continuous monitoring approaches

Safety standards:

• Risk management frameworks
• Safety assurance processes
• Incident response and reporting
• Governance and accountability
• Alignment verification

UK and international adoption:

• UK government procurement requirements
• Industry adoption (voluntary initially)
• International harmonization efforts
• Foundation for potential regulation
• Soft power and norm-setting

6. Policy Advice and Government Coordination

Advising UK government:

Policy development:

• Risk assessments for ministers and parliament
• Technical input to AI legislation
• Regulatory framework design
• International negotiation support
• Public communication strategies

Interagency coordination:

• National security agencies (GCHQ, MI5, MI6)
• Regulatory bodies (ICO, CMA, Ofcom)
• Research funders (UKRI)
• Science and technology departments
• Foreign policy (Foreign Office)

Parliamentary engagement:

• Expert testimony to committees
• Briefings for MPs
• Technical translation for policymakers
• Risk communication
• Policy option analysis

Key Personnel and Expertise

Ian Hogarth (Chair)

Background:

• Technology entrepreneur and investor
• AI safety advocate
• Wrote influential Financial Times op-ed on "God-like AI"
• Connections to UK tech sector and government
• Strong safety focus

Leadership:

• Strategic direction for AISI
• External relations and advocacy
• Government and international engagement
• Representing AISI publicly
• Ensuring independence and rigor

Approach:

• Taking AI risks seriously
• Building credible technical organization
• International cooperation
• Balancing innovation with safety
• Evidence-based policymaking

Staffing and Expertise

Hiring from:

• AI safety organizations (METR, Apollo, ARC, Anthropic, etc.)
• UK universities (Oxford, Cambridge, Imperial, etc.)
• Frontier AI labs (researchers concerned about safety)
• Other government departments (GCHQ, etc.)
• International talent

Expertise needed:

• Machine learning and AI systems
• Evaluation and red-teaming
• Cybersecurity and CBRN
• Interpretability and alignment
• Policy and governance
• International relations

Hiring challenges:

• Competing with private sector salaries (London is expensive)
• Government hiring processes
• Security clearances required for some roles
• Need for rapid scaling
• Retention in fast-moving field

Advantages:

• Mission-driven work with government backing
• Influence on policy and practice
• Access to frontier models
• International platform
• Job security and meaningful impact

Relationship with AI Labs

Cooperation framework:

Voluntary Agreements

Lab commitments:

• Provide pre-release access to models
• Share safety-relevant information
• Participate in evaluations
• Engage on standard-setting
• International coordination

AISI commitments:

• Responsible handling of sensitive information
• Constructive engagement
• Technical rigor
• Timely evaluations
• Security of proprietary data

Labs participating:

• OpenAI
• Anthropic
• Google DeepMind
• Others as they develop frontier models

Information Sharing

What labs share:

• Model access for evaluation (pre-deployment)
• Technical documentation
• Safety research findings
• Risk assessments
• Incident reports (if any)

Confidentiality:

• NDAs and secure handling
• Classified information protocols
• Public reporting (aggregated/redacted)
• National security considerations
• Competitive sensitivity

Mutual Benefits

For labs:

• Independent validation of safety claims
• Early warning of risks
• Inform better safety practices
• Government relationships
• Social license and legitimacy

For AISI:

• Access to frontier systems
• Understanding of frontier capabilities
• Influence on deployment decisions
• Technical learning
• Real-world testing

Tensions and Challenges

Potential conflicts:

• Labs want speed, AISI wants thoroughness
• Commercial sensitivity vs. transparency
• National security complications
• Regulatory uncertainty
• International competition

Future evolution:

• Currently voluntary, might become mandatory
• Regulatory framework could formalize
• International coordination might create pressure
• Labs might resist if too burdensome
• Balance between cooperation and oversight

International Leadership and Influence

AI Safety Summits

Bletchley Park Summit (November 2023):

• UK convened first international AI Safety Summit
• 28 countries and EU committed to AI safety cooperation
• Bletchley Declaration recognized catastrophic risks
• Foundation for international coordination
• UK positioned as leader

Second Summit (2024):

• South Korea and UK co-hosting
• Building on Bletchley commitments
• Deepening international cooperation
• Expanding participation
• Concrete safety measures

Ongoing convenings:

• Regular international meetings
• Technical working groups
• Research collaborations
• Standard harmonization
• Diplomatic engagement

Setting Global Norms

UK influence:

Standard-setting:

• Evaluation methodologies becoming international reference
• Safety frameworks adopted by other countries
• Best practices dissemination
• Training and capacity building
• Soft power through technical leadership

Bridging role:

• Between US and Europe
• Academic and government
• Technical and political
• Innovation and safety
• National and international

Challenges:

• China not participating fully
• Different national approaches
• Verification and enforcement
• Balancing openness and security
• Resource constraints

Impact and Accomplishments

Quantified Outcomes

On UK AI Policy

AISI provides technical expertise directly to Parliament and ministers, informing risk assessments for policy decisions and shaping the framework for UK AI regulation. Key policy inputs include the Online Safety Act AI provisions, AI White Paper consultation responses, regulatory framework development, and national security AI strategy.

On Industry Practices

AISI's evaluation methods are becoming industry standard. Major labs (OpenAI, Anthropic, Google DeepMind) have entered voluntary agreements for pre-deployment testing, with Inspect AI being adopted as the evaluation framework of choice. This has normalized pre-deployment safety assessment, increased transparency requirements, and established red-teaming as standard practice.

On International AI Safety

The UK is now recognized as a global leader in AI safety convening. Concrete outcomes include the Bletchley Declaration (first international agreement on AI risks), the International Network of AI Safety Institutes (10+ member countries), shared evaluation methodologies through Inspect AI, the International Scientific Report on AI Safety (led by Turing Award winner Yoshua Bengio), and foundation for ongoing governance discussions at summits in Seoul and Paris.

Criticisms and Challenges

The 2025 Rebrand Controversy

The February 2025 renaming from "AI Safety Institute" to "AI Security Institute" drew significant criticism from the AI safety community. AI Now Institute noted that the change signals the institute "will no longer focus on AI ethical issues, such as algorithm bias or protecting freedom of speech in AI applications." Critics worry this represents a narrowing of scope away from broader societal harms toward a purely national security focus.

Defenders argue the rebrand reflects AISI's actual priorities from the beginning, with Ian Hogarth emphasizing that "this change of name better reflects the team's essential remit and the priorities that they have always focused on."

Can Government Keep Pace?

AI capabilities advance faster than government institutions can respond. AISI faces structural challenges including bureaucratic constraints on hiring speed, difficulty matching private sector salaries (despite being competitive within civil service), limited compute resources compared to lab budgets, and inevitable evaluation lag behind the frontier.

AISI's response has been to operate as a "startup inside government" with fast-track hiring, partnerships with academic and private sector organizations, and focus on high-impact evaluation work rather than trying to match lab resources directly.

Regulatory Capture Risk

There are concerns that AISI might be captured by industry interests given its dependence on voluntary lab cooperation, the information asymmetry favoring labs, and the revolving door between government and industry. However, mitigations include government independence, diverse expertise sourcing (not only from labs), public interest mandate, parliamentary oversight, and increasing transparency.

International Coordination Challenges

China's absence from Western AI safety coordination remains a significant limitation. The International Network of AI Safety Institutes includes primarily democratic allies, with no clear pathway to engaging China despite early participation in the Bletchley Declaration. This creates risks of fragmented governance and potential race dynamics.

Resources and Scope

With an annual budget of approximately 50 million GBP, AISI remains smaller than major lab research budgets and must prioritize carefully. The Institute cannot evaluate everything and depends heavily on partnerships. Maintaining quality and credibility while scaling remains an ongoing challenge.

Future Directions

Near-Term (1-2 years)

Organizational development:

• Complete initial hiring and team building
• Establish robust evaluation infrastructure
• Deepen lab partnerships
• Expand international collaboration
• Demonstrate technical credibility

Key deliverables:

• Regular frontier model evaluations
• Published research and standards
• Second AI Safety Summit outcomes
• International coordination agreements
• Policy recommendations to UK government

Medium-Term (2-5 years)

Institutionalization:

• Mature organization with stable funding
• Recognized global leader in AI safety
• Influential in international standards
• Comprehensive evaluation program
• Growing team and capabilities

Possible developments:

• Regulatory authority (if legislation passed)
• Mandatory evaluation requirements
• International verification mechanisms
• Expanded scope (multimodal, robotics, etc.)
• Regional presence beyond London

Long-Term Vision

UK as global AI safety hub:

• Leading technical expertise
• Convening power for international cooperation
• Standards and norms shaping global governance
• Preventing catastrophic AI deployments
• Contributing to beneficial AI development

Broader impact:

• Effective international AI governance
• Safe development of transformative AI
• UK playing key role in existential risk reduction
• Model for other countries
• Public trust in AI governance

Comparisons to Other Organizations

vs US AI Safety Institute

Similarities:

• Both government AI safety bodies
• Similar missions and approaches
• Close coordination and partnership
• Created around same time (2023)

Differences:

• UK: Smaller budget and country, more nimble
• US: Larger resources, broader scope
• UK: Explicit international hub strategy
• US: Within NIST, different structure
• UK: More unified government approach
• US: More complex federal system

Relationship: Sister organizations, very close collaboration

vs Private Safety Organizations

Complementary:

• METR, Apollo, ARC: Private orgs focused on research and evaluation
• UK AISI: Government body with policy authority
• AISI learns from and hires from safety orgs
• Different authorities and mandates
• Collaborative ecosystem

vs Frontier Labs

Different roles:

• Labs: Build AI systems, commercial
• AISI: Evaluate and oversee, government
• Labs: Innovation focus
• AISI: Safety focus
• Cooperative but some tension
• AISI provides independent assessment

Key Developments Timeline

Sources and Resources

Official Sources

• UK AI Security Institute Official Website↗🏛️ government★★★★☆UK AI Safety InstituteAI Safety Institutesafetysoftware-engineeringcode-generationprogramming-ai+1Source ↗ - Main portal for the Institute
• GOV.UK AI Safety Institute Page↗🏛️ government★★★★☆UK GovernmentUK AISIcapabilitythresholdrisk-assessmentgame-theory+1Source ↗ - Government organizational page
• Introducing the AI Safety Institute↗🏛️ government★★★★☆UK GovernmentIan Hogarthgovernancegovernment-ai-safetyinternationalSource ↗ - Official overview document
• Fourth Progress Report↗🏛️ government★★★★☆UK AI Safety InstituteFourth Progress Reportgovernancegovernment-ai-safetyinternationalSource ↗ - May 2024 progress update
• Our 2025 Year in Review↗🏛️ government★★★★☆UK AI Safety InstituteOur 2025 Year in Reviewgovernancegovernment-ai-safetyinternationalSource ↗ - 2025 annual review

Research Outputs

• Frontier AI Trends Report↗🏛️ government★★★★☆UK AI Safety InstituteAISI Frontier AI TrendsA comprehensive government assessment of frontier AI systems shows exponential performance improvements in multiple domains. The report highlights emerging capabilities, risks, ...capabilitiessafetybenchmarksred-teaming+1Source ↗ - December 2024 assessment of frontier AI capabilities
• Inspect AI Framework↗🔗 webUK AI Safety Institute's Inspect frameworksafetyinterpretabilityscalable-oversightrlhfSource ↗ - Open-source evaluation framework documentation
• Inspect Evals↗🔗 webInspect EvalsevaluationSource ↗ - Collection of 100+ pre-built evaluations
• Early Lessons from Evaluating Frontier AI Systems↗🏛️ government★★★★☆UK AI Safety InstituteUK AI Security Institute's evaluationsevaluationcybersecuritySource ↗ - Evaluation methodology insights

Funding Programs

• The Alignment Project↗🏛️ governmentThe Alignment ProjectalignmentSource ↗ - 15 million GBP global alignment research fund
• Grants Overview↗🏛️ government★★★★☆UK AI Safety InstituteGrants Overviewgovernancegovernment-ai-safetyinternationalSource ↗ - Information on all AISI grant programs
• Systemic Safety Grants↗🏛️ government★★★★☆UK AI Safety InstituteSystemic Safety GrantssafetySource ↗ - 8.5 million GBP societal resilience program

International Coordination

• The Bletchley Declaration↗🏛️ government★★★★☆UK Governmentgovernment AI policiesx-riskeffective-altruismlongtermismgame-theory+1Source ↗ - November 2023 international agreement
• Chair's Summary of AI Safety Summit 2023↗🏛️ government★★★★☆UK GovernmentEarly stagesrisk-factordiffusioncontrolSource ↗ - Summit outcomes
• CSIS: The AI Safety Institute International Network↗🔗 web★★★★☆CSISThe AI Safety Institute International Network: Next StepssafetySource ↗ - Analysis of international coordination
• Centre for Future Generations: The AI Safety Institute Network↗🔗 webCentre for Future Generations: The AI Safety Institute NetworksafetySource ↗ - Comparative analysis

News and Analysis

• UK AI Safety Institute Wikipedia↗📖 reference★★★☆☆WikipediaUK AI Safety Institute WikipediasafetySource ↗ - Comprehensive background
• Computer Weekly: San Francisco Office Announcement↗🔗 webComputer Weekly: San Francisco Office AnnouncementcomputeSource ↗ - May 2024 expansion
• Infosecurity Magazine: AISI Rebrands↗🔗 webInfosecurity Magazine: AISI Rebrandsgovernancegovernment-ai-safetyinternationalSource ↗ - February 2025 name change
• AI Now Statement on Transition↗🔗 webAI Now Statement on Transitiongovernancegovernment-ai-safetyinternationalSource ↗ - Critical perspective on rebrand