Governance.ai: Open Problems in Technical AI Governance

government

Centre for the Governance of AI·cdn.governance.ai/Open_Problems_in_Technical_AI_Governanc...

Credibility Rating

4/5

High(4)

High quality. Established institution or organization with editorial oversight and accountability.

Rating inherited from publication venue: Centre for the Governance of AI

Published by Governance.ai, this paper is a key reference for researchers seeking to understand where technical AI work can most meaningfully support real-world AI governance and policy implementation.

Metadata

Importance: 78/100organizational reportanalysis

Summary

This paper defines and scopes 'technical AI governance' (TAIG) as the application of technical analysis and tools to support effective AI governance. It presents a structured taxonomy of open problems organized around three functions: identifying where governance intervention is needed, assessing the efficacy of governance actions, and designing enforcement and compliance mechanisms. The work is intended to guide technical researchers and funders toward high-impact contributions at the intersection of AI policy and technical research.

Key Points

•Introduces 'technical AI governance' (TAIG) as a distinct field bridging technical AI research and policy/governance needs.
•Organizes open problems into three core functions: identifying intervention needs, assessing governance efficacy, and designing enforcement/compliance mechanisms.
•Highlights a critical gap: decision-makers lack sufficient technical information to identify risks and evaluate governance options.
•Serves as a prioritization resource for researchers and funders seeking to direct effort toward policy-relevant technical problems.
•Covers topics including model evaluation, monitoring, auditing, compute governance, and standards development.

Cached Content Preview

HTTP 200Fetched Mar 15, 202698 KB

# Open Problems in Technical AI Governance

Anka Reuel∗ Stanford University Ben Bucknall $^ \*$ Centre for the Governance of AI & Oxford Martin AI Governance Initiative

[anka.reuel@stanford.edu](mailto:anka.reuel@stanford.edu) [ben.bucknall@governance.ai](mailto:ben.bucknall@governance.ai)

Stephen Casper MIT CSAIL

Tim Fist Institute for Progress & Center for a New American Security Lisa Soder interface – Tech Analysis and Policy Ideas for Europe e.V. Onni Aarne Institute for AI Policy and Strategy

Lewis Hammond University of Oxford & Cooperative AI Foundation Lujain Ibrahim University of Oxford

Alan Chan Centre for the Governance of AI & Mila

Peter Wills Centre for the Governance of AI & University of Oxford Markus Anderljung Centre for the Governance of AI

Ben Garfinkel Centre for the Governance of AI

Lennart Heim Centre for the Governance of AI

Andrew Trask OpenMined & University of Oxford

Gabriel Mukobi Stanford University

Rylan Schaeffer Stanford University

Mauricio Baker Independent Researcher

Sara Hooker Cohere For AI

Irene Solaiman Hugging Face

Alexandra Sasha Luccioni Hugging Face

Nitarshan Rajkumar University of Cambridge

Nicolas Moës The Future Society

Neel Guha Stanford University

Jessica Newman University of California, Berkeley

Yoshua Bengio University of Montreal & Mila

Tobin South MIT

Alex Pentland Stanford HAI

Jeffrey Ladish Palisade Research

Sanmi Koyejo Stanford University

Mykel J. Kochenderfer Stanford University

Robert Trager Oxford Martin AI Governance Initiative, Blavatnik Sch University of Oxford

# Abstract

AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the barriers and uncertainties faced are at least partly technical. Technical AI governance, referring to technical analysis and tools for supporting the effective governance of AI, seeks to address such challenges. It can help to (a) identify areas where intervention is needed, (b) identify and assess the efficacy of potential governance actions, and (c) enhance governance options by designing mechanisms for enforcement, incentivization, or compliance. In this paper, we explain what technical AI governance is, why it is important, and present a taxonomy and incomplete catalog of its open problems. This paper is intended as a resource for technical researchers or research funders looking to contribute to AI governance.

# 1 Introduction

The rapid development and adoption of artificial intelligence (AI) systems $^ { 1 }$ has prompted a great deal of governance action from the public sector $^ 2$ , academia and civil society (Anderljung et al., 2023a; Moës & Ryan, 2023; Barrett et al., 2023), and industry (Anthropic, 2023a; Microsoft, 2023; Dragan et al., 2024; OpenAI, 2024a), with the aim of addressing potential risks while capitalising on benefits.

However, key decision-makers seeking to govern AI often have insufficient information for identifying the need

... (truncated, 98 KB total)

Resource ID: b4402422d8628b71 | Stable ID: ZGM2Y2E0NG