Skip to content
Longterm Wiki

Brief Analysis Of Op Technical Ai Safety Funding

blog
LessWrong·lesswrong.com/posts/adzfKEW98TswZEA6T/brief-analysis-of-o...

Credibility Rating

3/5
Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: LessWrong

A LessWrong analysis of Open Philanthropy's funding patterns in technical AI safety, useful for understanding resource allocation in the field and which organizations and research directions receive philanthropic support.

Forum Post Details

Karma
82
Comments
5
Forum
lesswrong
Forum Tags
AICommunity

Metadata

Importance: 45/100blog postanalysis

Summary

This post analyzes Open Philanthropy's grantmaking in technical AI safety, examining funding patterns, recipient organizations, and how resources are allocated across different research approaches. It provides a data-driven look at how one of the field's major funders distributes its technical AI safety budget.

Key Points

  • •Examines the distribution of Open Philanthropy's technical AI safety grants across organizations and research areas
  • •Provides quantitative breakdown of funding amounts and recipient types in the technical AI safety ecosystem
  • •Identifies which research directions and institutions receive the most philanthropic support
  • •Highlights potential gaps or concentration risks in how technical AI safety funding is allocated
  • •Offers context for understanding the financial landscape shaping AI safety research priorities

1 FactBase fact citing this source

EntityPropertyValueAs Of
Coefficient GivingDescription68% of technical AI safety grants went to evaluations/benchmarking (2024)—

Cached Content Preview

HTTP 200Fetched Apr 7, 20264 KB
# Brief analysis of OP Technical AI Safety Funding
By 22tom
Published: 2024-10-25
**TL;DR**
---------

I spent a few hours going through Open Philanthropy (OP)'s grant database. The main findings were:

*   Open Philanthropy has made $28 million grants for Technical AI Safety (TAIS) in 2024
    *   68% of these are focused on evaluations / benchmarking. The rest is split between interpretability, robustness, value alignment, forecasting, field building and other approaches.
*   OP funding for TAIS has fallen from a peak in 2022
    *   Excluding funding for evaluations, TAIS funding has fallen by ~80% since 2022.
*   A majority of TAIS funding is focused on "meta" rather than "direct" safety approaches

My overall takeaway was that very few TAIS grants are directly focused on making sure systems are aligned / controllable / built safely.[^j5qxrejbakh] 

**Method**
----------

I:

1.  Downloaded [OP's list of grants](https://www.openphilanthropy.org/grants/)
2.  Filtered for "Potential Risks from Advanced AI"
3.  Classified grants as either "Policy" or "Non-Policy"
4.  Within "Non-Policy", classified grants by different focus areas (e.g. evaluations, interpretability, Field building, "Multiple" and "Other")
    1.  In most cases I classified just from the grant name; occasionally I dug for a bit more info. There are definitely errors, and cases where grants could be more clearly specifcied.
5.  Combined focus areas into "Clusters" - "Empirical", "Theory", "Forecasting & Evaluating", "Fieldbuilding" and "Other"
6.  Created charts 

**Results**
-----------

**Grants by Research Agenda **

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/028ed847ed8d956255a8a3e9608245a2ae97aa9b0bb807e2.png)

**Grants by Cluster**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1cec53f4ba0b89a3a9d8ba249b2506e78a2f59060aaff3d4.png)

Full data available [here](https://docs.google.com/spreadsheets/d/1L_KAlLhXLJuPPD4XQlpvxi_scLRUDIKmiUJodQntGYY/copy)

Key Findings
------------

**(1) Evaluations & Benchmarking make up 2/3rds of all OP TAIS funding in 2024**

Most of these grants are related to the [RFP on LLM Benchmarks](https://www.openphilanthropy.org/rfp-llm-benchmarks/).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/873b844f013db87178f40c7559465a308b2e8ef7cd9dd0cc.png)

TAIS Grants in 2024 by Research Agenda

**(2) Excluding Evaluations & Benchmarking, OP grants for TAIS have fallen significantly**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/98c1e1004f8c57386d2411b6fdcdba5f5bf8159fe3d6ef88.png)

*   2022 Funding (excluding evaluations): $62,089,504
*   2023 Funding (excluding evaluations): $43,417,089
*   2024 Funding (projected, excluding evaluations): $10,808,390
    *   82.6% reduction vs 2022

**(3) Most TAIS funding is focused on "investment" rather than direct approaches to AI safety**

I classify grants into two broad buckets:

1.  "Direct" - grants for research agendas which aim to improve safety toda

... (truncated, 4 KB total)
Resource ID: kb-fb66d73671ec9ced | Stable ID: sid_BHOece3h1n