Back
Institute for AI Policy and Strategy analysis
webCredibility Rating
4/5
High(4)High quality. Established institution or organization with editorial oversight and accountability.
Rating inherited from publication venue: Institute for AI Policy and Strategy
Published by the Institute for AI Policy and Strategy (IAPS), this resource is particularly useful for those wanting a cross-company landscape view of where technical safety research is happening in industry, relevant for both governance actors and researchers assessing field coverage.
Metadata
Importance: 62/100organizational reportanalysis
Summary
An IAPS analysis that maps and categorizes the technical AI safety research being conducted across major AI companies, identifying what areas are being prioritized, where gaps exist, and how industry research agendas compare. It provides a structured overview of the technical safety landscape within frontier AI labs.
Key Points
- •Surveys and categorizes technical safety research efforts across major AI companies including OpenAI, Anthropic, DeepMind, and others
- •Identifies which safety research areas (interpretability, alignment, robustness, etc.) are receiving attention and which may be underinvested
- •Enables comparison of research priorities across organizations to highlight convergences and gaps in the field
- •Useful for policymakers and researchers seeking to understand the state of industry-led technical safety work
- •Bridges governance and technical safety communities by making lab research legible to policy audiences
Cited by 2 pages
| Page | Type | Quality |
|---|---|---|
| AI Safety Technical Pathway Decomposition | Analysis | 62.0 |
| Corporate Influence on AI Policy | Crux | 66.0 |
Cached Content Preview
HTTP 200Fetched Mar 20, 20263 KB
[0](https://www.iaps.ai/cart)
# Mapping Technical Safety Research at AI Companies
Sep 12, 2024
Written By [Oscar Delaney](https://www.iaps.ai/research?author=66e440472b468f4874056194)
[Read the full report](https://www.iaps.ai/s/Mapping-technical-safety-research.pdf)
As artificial intelligence (AI) systems become more advanced, concerns about large-scale risks from misuse or accidents have grown. This report analyzes the technical research into safe AI development being conducted by three leading AI companies: Anthropic, Google DeepMind, and OpenAI.
We define “safe AI development” as developing AI systems that are unlikely to pose large-scale misuse or accident risks. This encompasses a range of technical approaches aimed at ensuring AI systems behave as intended and do not cause unintended harm, even as they are made more capable and autonomous.
We analyzed all papers published by the three companies from January 2022 to July 2024 that were relevant to safe AI development, and categorized the 80 included papers into nine safety approaches. Additionally, we noted two categories representing nascent approaches explored by academia and civil society, but not currently represented in any research papers by these leading AI companies. Our analysis reveals where corporate attention is concentrated and where potential gaps lie.
Some AI research may stay unpublished for good reasons, such as to not inform adversaries about the details of safety and security techniques they would need to overcome to misuse AI systems. Therefore, we also considered the incentives that AI companies have to research each approach, regardless of how much work they have published on the topic. In particular, we considered reputational effects, regulatory burdens, and to what extent the approaches could be used to make the company’s AI systems more useful.
We identified three categories where there are currently no or few papers and where we do not expect AI companies to become much more incentivized to pursue this research in the future. These are model organisms of misalignment, multi-agent safety, and safety by design. Our findings provide an indication that these approaches may be slow to progress without funding or efforts from government, civil society, philanthropists, or academia.
_Note:_ _Due to a coding issue, an earlier version of this report excluded some relevant papers. We apologize for the error. This report is the corrected version, finalized on September 25th, 2024._
[Policy and Standards](https://www.iaps.ai/research/tag/Policy+and+Standards)[Oscar Delaney](https://www.iaps.ai/research/tag/Oscar+Delaney)
[Oscar Delaney](https://www.iaps.ai/research?author=66e440472b468f4874056194)
[Previous\\
\\
Previous\\
**Response to the DOD RFI on Defense Industrial Base Adoption of Artificial Intelligence for Defense Applications**](https://www.iaps.ai/research/dod-dib-rfi-response) [Next\\
\\
Next\\
\\
**The Future of International Scientific Assessments of
... (truncated, 3 KB total)Resource ID:
c4fbe78110edcfab | Stable ID: OTJkMmJiMT