AI Safety Field Building

A retrospective on the MATS (ML Alignment Theory Scholars) Spring 2024 Extension program, reviewing outcomes, lessons learned, and the effectiveness of training researchers for AI safety work. It likely covers participant projects, program structure, and recommendations for future cohorts.

This report documents the outcomes and impact of ARENA 4.0, a technical AI safety training program designed to upskill researchers in alignment-relevant skills such as mechanistic interpretability and reinforcement learning. It provides data on participant outcomes, career trajectories, and the program's contribution to the AI safety talent pipeline.

This research piece from Coefficient Giving argues that AI safety and security research is significantly underfunded relative to the risks involved, and makes the case for philanthropists and funders to increase financial support for the field. It examines funding gaps, highlights promising organizations and research areas, and encourages diversification of the funder base beyond a few major donors.

BlueDot Impact's analysis of their 2022 AI Safety Fundamentals: Alignment course shows it increased the proportion of participants working on AI safety from 5% to 37% (342 participants), representing a net gain of 105 individuals entering the field. The post distinguishes between participants redirected into AI safety careers (20 people) versus those accelerated into roles they already planned for (85 people). BlueDot also notes significant improvements made to curriculum, tracking, and facilitator recruitment since 2022.

This page references an Open Philanthropy grant to MATS (Machine Learning Alignment Theory Scholars) program to cover AI safety research expenses. MATS is a field-building initiative that trains and supports emerging AI safety researchers. The grant reflects Open Philanthropy's ongoing investment in developing human capital for the AI safety field.

This article provides a comprehensive overview of AI Safety Institutes (AISIs) as a novel global governance model, cataloguing existing institutes worldwide and analyzing their core functions: evaluating frontier AI systems, conducting safety research, and facilitating stakeholder information exchange. It examines the historical development from the UK's 2023 Bletchley Park summit through a growing second wave of national institutes, and questions the recent shift in some jurisdictions from 'safety' to 'security' framing.

This EA Forum post argues that AI safety's talent pipeline is structurally biased toward producing researchers, despite leadership consensus that research is not the most neglected role. The author identifies feedback loops where research-centric programs disadvantage non-researchers in hiring, and calls for ecosystem-level coordination to better allocate talent across leadership, policy, and advocacy roles.

The AI Safety Fund (AISF) is a $10 million+ collaborative initiative launched in October 2023 by Anthropic, Google, Microsoft, and OpenAI (via the Frontier Model Forum) along with philanthropic partners to fund independent AI safety and security research. It has distributed two rounds of grants focused on responsible frontier AI development, public safety risk reduction, and standardized third-party capability evaluations. The fund is now directly managed by the Frontier Model Forum following the closure of its original administrator, the Meridian Institute.

80,000 Hours provides a comprehensive career guide for technical AI safety research, covering empirical and theoretical paths, entry requirements, key organizations, and strategies for building relevant skills. It frames AI safety research as one of the highest-impact career choices given existential risks from advanced AI systems.

A quantitative analysis tracking growth of the AI safety field from 2010–2025, estimating total FTEs grew from ~400 in 2022 to ~1,100 in 2025 across 119 organizations. Technical AI safety grew at 21% annually in FTEs, while non-technical (policy/governance) work expanded dramatically, now nearly matching technical headcount.

Open Philanthropy reviews its 2024 philanthropic activities and outlines priorities for 2025, with emphasis on AI safety research funding, strategic partnerships, and grants spanning global health and catastrophic risk reduction. The report provides transparency into one of the field's largest funders and signals where major resources will flow in the AI safety ecosystem.

80,000 Hours updates its research and recommendations regarding AI risk and career paths in AI safety, reflecting evolving views on the urgency and tractability of AI-related existential risks. The post outlines revised thinking on how individuals can best contribute to reducing AI risks through career choices, and adjusts priority areas based on current landscape assessments.

A curated guide from 80,000 Hours providing resources for individuals looking to develop technical skills relevant to AI safety research. It aggregates learning materials, courses, and pathways to help people transition into or advance within the technical AI safety field. The resource supports field-building by lowering barriers to entry for aspiring AI safety researchers.

A comprehensive survey of the AI safety funding landscape as of mid-2023, cataloging major philanthropic sources including Open Philanthropy, the FTX Future Fund, and the Long-Term Future Fund. The post maps the distribution of financial resources across AI safety research mechanisms and identifies key institutional players shaping the field's financial ecosystem.

The Center for AI Safety (CAIS) 2024 Impact Report summarizes the organization's activities, accomplishments, and reach over the year, covering its research, educational programs, field-building initiatives, and policy engagement. It provides an overview of how CAIS is working to reduce societal-scale risks from advanced AI systems.

Open Philanthropy issued a request for proposals seeking technical AI safety research projects, signaling funding priorities and research directions the organization considers most valuable. The RFP outlines areas of interest including interpretability, scalable oversight, and related alignment challenges, aiming to grow the field by supporting researchers and organizations working on these problems.

The Future of Life Institute evaluated eight major AI companies across 35 safety indicators, finding widespread deficiencies in risk management and existential safety practices. Even top performers Anthropic and OpenAI received only marginal passing grades, highlighting systemic gaps across the industry in preparedness for advanced AI risks.

CHAI is a UC Berkeley research center dedicated to reorienting AI development toward systems that are provably beneficial and aligned with human values. It conducts technical and conceptual research on problems including value alignment, corrigibility, and AI safety, and serves as a major hub for academic AI safety work.

ARENA (Alignment Research Engineer Accelerator) is an educational program designed to train technical AI safety researchers by providing structured curriculum covering mechanistic interpretability, reinforcement learning, and other core alignment topics. It aims to build the pipeline of competent AI safety engineers by offering hands-on, project-based learning. The program serves as a key field-building initiative to address the talent gap in technical AI safety research.

BlueDot Impact is an organization focused on building the AI safety field through structured educational programs and courses. It offers cohort-based training programs designed to help professionals transition into or contribute to AI safety and biosecurity work. Its programs aim to rapidly grow the pipeline of skilled talent working on existential risk reduction.

A landmark international scientific assessment co-authored by 96 experts from 30 countries, providing a comprehensive overview of general-purpose AI capabilities, risks, and risk management approaches. It aims to establish shared scientific understanding across nations as a foundation for global AI governance. The report covers topics including capability evaluation, misuse risks, systemic risks, and mitigation strategies.

ARENA 5.0 is a 4-week intensive in-person program that upskills technically talented individuals for AI safety work, covering mechanistic interpretability, reinforcement learning, and LLM evaluations. The fifth cohort of 28 participants achieved the highest satisfaction rating to date (9.3/10), with 8 participants securing confirmed offers in technical AI safety roles.

MATS is an intensive fellowship program designed to help researchers transition into AI safety careers, offering structured mentorship from leading researchers, stipends, and community integration. Since 2021, it has trained over 446 researchers who have collectively produced 150+ research papers and gone on to work at top AI safety organizations.

Catalyze Impact is a nonprofit incubator addressing organizational bottlenecks in the AI safety ecosystem by connecting founders with co-founders, mentors, and funding. Their pilot program incubated 11 organizations from 200+ applicants, with participants reporting an average 11-month acceleration in progress. They are hiring two key roles to scale their operations.

A searchable database of Open Philanthropy grants related to AI safety, providing transparency into one of the field's largest funders. It documents funding awarded to universities, research organizations, and individuals working on technical and governance aspects of AI safety. This resource helps map the landscape of institutional support for AI safety research.

A longitudinal study tracking the growth of the AI safety field from 2010 to 2025, documenting expansion from approximately 400 to 1,100 full-time equivalent researchers across both technical and non-technical domains. The analysis provides quantitative benchmarks for understanding the field's scaling trajectory and workforce development over 15 years.

Open Philanthropy is a major philanthropic organization that funds work across global health, AI safety, biosecurity, and other cause areas. Their grants database provides transparency into which organizations and research directions receive funding. They are one of the largest funders of AI safety and existential risk research.

SPAR (Supervised Program for Alignment Research) is a structured mentorship program that pairs aspiring researchers with experienced AI safety professionals to conduct research on AI safety, alignment, and policy topics. The program provides hands-on research experience, guidance from domain experts, and opportunities for publication, serving as an entry point for newcomers to the AI safety field.

The Future of Life Institute's AI Safety Index 2024 systematically evaluates six leading AI companies—including OpenAI, Google DeepMind, Anthropic, Meta, xAI, and Mistral—across 42 safety indicators spanning risk management, transparency, governance, and preparedness for advanced AI threats. The index finds widespread deficiencies in safety practices and provides letter-grade assessments to benchmark industry progress. It serves as a comparative accountability tool aimed at pressuring companies toward stronger safety commitments.

This report introduces the Technical Alignment Research Accelerator (TARA), a 14-week part-time program designed to fill the gap between introductory AI safety awareness and elite full-time research fellowships. TARA's inaugural cohort demonstrated strong outcomes including a 9.43/10 recommendation score, 90% completion rate, and only $899 AUD cost per participant. The authors argue this accessible, remote-friendly model can expand the AI safety talent pipeline by removing barriers of relocation, career interruption, and financial constraint.