AI Safety Talent Supply/Demand Gap Model
AI Safety Researcher Gap Model
Quantifies AI safety talent shortage: current 300-800 unfilled positions (30-50% gap) with training pipelines producing only 220-450 researchers annually when 500-1,500 are needed. Projects gaps could worsen to 50-60% by 2027 in scaling scenarios, with A-tier researcher shortage (50-100 vs 200-400 needed) particularly critical; recommends tripling MATS-style programs ($45M), competitive salary funds ($50-100M/year), and new PhD programs ($40-80M) with estimated 4-9x ROI.
Overview
This model analyzes the persistent mismatch between AI safety researcher supply and organizational demand, with critical implications for alignment research progress timelines. The analysis reveals a structural talent shortage that represents one of the most binding constraints on AI safety progress.
Current estimates show 300-800 unfilled safety research positions (30-50% of total demand), with training pipelines producing only 220-450 qualified researchers annually when 500-1,500 are needed. Under scaling scenarios where AI safety becomes prioritized, this gap could expand to 50-60% by 2027, fundamentally limiting the field's ability to address alignment difficulty before advanced systems deployment.
The model identifies four critical bottlenecks: insufficient training pathways, funding constraints, coordination failures, and competing demand from capabilities development, with intervention analysis suggesting targeted programs could cost-effectively expand supply.
Risk Assessment
| Dimension | Assessment | Evidence | Timeline |
|---|---|---|---|
| Severity | Critical - talent shortage limits all safety progress | 3-10x gap between needed and available researchers | Ongoing |
| Likelihood | Very High - structural problem worsening | 70-90% probability gap persists under AI scaling | 2025-2030 |
| Trend | Negative - gap widening faster than solutions | Pipeline growth 15-25%/year vs demand growth 30-100%/year | Deteriorating |
| Tractability | Medium-High - proven interventions available | MATS-style programs show 60-80% placement rates | Immediate opportunities |
Current Supply Analysis
Narrow Definition Supply (Technical AI Safety)
| Category | 2024 Estimate | Growth Rate | Quality Distribution |
|---|---|---|---|
| Full-time technical researchers | 300-500 | 20%/year | 20% A-tier, 50% B-tier, 30% C-tier |
| Safety-focused PhD students | 200-400 | 25%/year | 30% A-tier potential |
| Lab safety engineers | 500-1,000 | 30%/year | 10% A-tier, 60% B-tier |
| Total narrow supply | 1,000-1,900 | 25%/year | 15% A-tier overall |
Broader Definition Supply (Safety-Adjacent)
Organizations like Anthropic, OpenAI, and DeepMind employ researchers working on safety-relevant problems who don't identify primarily as safety researchers.
| Category | 2024 Estimate | Conversion Rate to Safety |
|---|---|---|
| ML researchers with safety interest | 2,000-5,000 | 5-15% |
| Interpretability/robustness researchers | 1,000-2,000 | 20-40% |
| AI governance/policy researchers | 500-1,000 | 10-30% |
| Potential conversion pool | 3,500-8,000 | 10-25% |
Demand Assessment
Current Organizational Demand (2024)
| Organization Type | Open Positions | Fill Rate | Salary Range | Source |
|---|---|---|---|---|
| Frontier labs (safety teams) | 500-1,000 | 60-80% | $150-800K | Anthropic careers↗🔗 web★★★★☆AnthropicAnthropic CareersThis is Anthropic's recruitment portal; relevant to those interested in joining the AI safety field or understanding how a leading safety-focused lab articulates its culture and principles to prospective employees.Anthropic's careers page outlines the company's mission to build safe and beneficial AI, highlighting their guiding principles and inviting researchers, engineers, and builders ...ai-safetyalignmentfield-buildingtalent+3Source ↗, OpenAI jobs↗🔗 web★★★★☆OpenAIOpenAI Careers PageThis is OpenAI's job listings page, useful as a reference for understanding hiring trends at a major frontier AI lab and for those exploring career paths in AI safety or related fields.The OpenAI careers page lists open positions across research, engineering, policy, and operations roles at OpenAI. It serves as a recruiting hub for one of the leading AI labs, ...field-buildingtalentcapabilitiesai-safety+2Source ↗ |
| Academic safety groups | 200-400 | 40-60% | $80-200K | University job boards |
| Safety orgs (MIRI, CHAI, etc.) | 100-200 | 50-70% | $100-300K | 80,000 Hours job board |
| Government/policy roles (AISI) | 50-100 | 30-50% | $120-250K | USAjobs.gov |
| Total current demand | 850-1,700 | 50-70% | Varies | Multiple sources |
Projected Demand Under Scaling Scenarios
| Scenario | Description | 2027 Demand | Demand Multiple |
|---|---|---|---|
| Baseline | Current growth trajectory | 1,300-2,500 | 1.5x |
| Moderate Scaling | Safety becomes industry priority | 2,500-5,000 | 3x |
| Crisis Response | Government/industry mobilization | 4,000-17,000 | 5-10x |
| Manhattan Project | Wartime-level resource allocation | 10,000-30,000 | 12-18x |
Training Pipeline Bottlenecks
Pipeline Capacity Analysis
The training pipeline represents the most significant constraint on talent supply, with current pathways producing insufficient researchers to meet projected demand.
| Training Pathway | Annual Output | Time to Competence | Quality Level | Cost per Researcher |
|---|---|---|---|---|
| PhD programs (safety-focused) | 20-50 | 4-6 years | High | $200-400K total |
| MATS-style programs | 50-100 | 6-12 months | Medium-High | $30-50K |
| Self-study/independent | 100-200 | 1-3 years | Variable | $10-30K |
| Industry transition programs | 50-100 | 1-2 years | Medium | $50-100K |
| Total pipeline capacity | 220-450/year | 1-6 years | Mixed | $30-400K |
Pipeline Efficiency Metrics
Current training programs show significant variation in effectiveness and cost-efficiency:
| Program | Completion Rate | Placement Rate | Cost Efficiency | Success Factors |
|---|---|---|---|---|
| MATS↗🔗 webMATS Research ProgramMATS is one of the primary talent pipelines into the AI safety field; wiki users interested in career transitions or field-building efforts should consider this a key institutional reference.MATS is an intensive fellowship program designed to help researchers transition into AI safety careers, offering structured mentorship from leading researchers, stipends, and co...ai-safetyalignmentfield-buildingeducational+4Source ↗ | 85-90% | 70-80% | High | Mentorship, practical projects |
| SERI MATS | 80-85% | 60-70% | High | Research experience |
| PhD programs | 70-80% | 90-95% | Medium | Deep expertise, credentials |
| Bootcamps | 60-70% | 40-60% | Medium | Intensive format |
Bottleneck Deep Dive
Bottleneck 1: Training Pipeline Constraints
Problem: Current training capacity produces only 30-50% of needed researchers annually.
Quantitative Breakdown:
- Required new researchers (to close gap by 2027): 500-1,500/year
- Current pipeline output: 220-450/year
- Pipeline deficit: 50-1,050/year (55-70% shortfall)
Quality Distribution Issues:
- A-tier researchers needed: 200-400
- A-tier production: 50-100/year
- A-tier gap: 100-300 (50-75% of demand)
Bottleneck 2: Funding Architecture
Organizations like Coefficient Giving↗🔗 web★★★★☆Coefficient GivingOpen Philanthropy grants databaseOpen Philanthropy is one of the most influential funders in AI safety; their grants database is a useful reference for understanding which organizations and research directions receive major philanthropic support.Open Philanthropy is a major philanthropic organization that funds work across global health, AI safety, biosecurity, and other cause areas. Their grants database provides trans...ai-safetyexistential-riskgovernancecoordination+3Source ↗ provide substantial funding, but total resources remain insufficient for scaling scenarios.
| Funding Source | 2024 Allocation | Growth Rate | Sustainability |
|---|---|---|---|
| Coefficient Giving | $50-100M | Stable | Medium-term |
| Frontier lab budgets | $100-300M | 20-30%/year | Market-dependent |
| Government funding | $20-50M | Slow | Policy-dependent |
| Other foundations | $10-30M | Variable | Uncertain |
| Total funding | $180-480M | 15-25%/year | Mixed |
Bottleneck 3: Competition from Capabilities Research
The racing dynamics between safety and capabilities create severe talent competition, with capabilities roles offering substantially higher compensation.
| Experience Level | Safety Org Salary | Capabilities Lab Salary | Premium Ratio |
|---|---|---|---|
| Entry-level | $80-120K | $200-400K | 2-3x |
| Mid-level | $120-200K | $400-800K | 3-4x |
| Senior | $200-300K | $600K-2M+ | 3-7x |
| Leadership | $250-400K | $1M-10M+ | 4-25x |
Intervention Analysis
High-Impact Training Interventions
| Intervention | Annual Cost | Output Increase | Cost per Researcher | Implementation Timeline |
|---|---|---|---|---|
| Scale MATS programs 3x | $15-30M | +200/year | $75-150K | 6-12 months |
| New safety PhD programs | $40-80M | +80/year | $500K-1M | 2-3 years |
| Industry transition bootcamps | $20-40M | +100-200/year | $100-200K | 6-12 months |
| Online certification programs | $5-10M | +100-300/year | $17-100K | 3-6 months |
Retention and Quality Interventions
Current annual attrition rates of 16-32% represent significant talent loss that could be reduced through targeted interventions.
| Retention Strategy | Cost | Attrition Reduction | ROI Analysis |
|---|---|---|---|
| Competitive salary fund | $50-100M/year | 5-10 percentage points | 2-4x researcher replacement cost |
| Career development programs | $10-20M/year | 3-5 percentage points | 3-5x |
| Research infrastructure | $20-40M/year | 2-4 percentage points | 2-3x |
| Geographic flexibility | $5-10M/year | 2-3 percentage points | 4-6x |
Scenario Modeling
Baseline Scenario: Current Trajectory
Under current trends, the talent gap improves modestly but remains significant:
| Year | Supply | Demand | Gap | Gap % |
|---|---|---|---|---|
| 2024 | 1,500 | 1,300 | -200 | 15% |
| 2025 | 1,800 | 1,600 | -200 | 13% |
| 2026 | 2,100 | 2,000 | -100 | 5% |
| 2027 | 2,500 | 2,800 | +300 | 11% |
Crisis Response Scenario
If AI progress triggers safety prioritization, gaps could become critical:
| Year | Supply (Enhanced) | Demand (Crisis) | Gap | Gap % |
|---|---|---|---|---|
| 2024 | 1,500 | 1,300 | -200 | 15% |
| 2025 | 2,200 | 3,000 | +800 | 27% |
| 2026 | 3,500 | 7,000 | +3,500 | 50% |
| 2027 | 6,000 | 15,000 | +9,000 | 60% |
Historical Precedents
Manhattan Project Comparison
The Manhattan Project↗🔗 webManhattan Project - Atomic Heritage FoundationUsed in AI safety discourse as a historical analogy for transformative, dual-use technology development under urgency; relevant to discussions of talent mobilization, governance, and the ethics of powerful technology.An overview of the Manhattan Project, the large-scale US government program during WWII that developed the first nuclear weapons. It serves as a historical case study of rapid, ...governanceexistential-riskcoordinationpolicy+4Source ↗ provides insights into rapid scientific talent mobilization:
| Metric | Manhattan Project (1942-1945) | AI Safety (Current) | AI Safety (Mobilized) |
|---|---|---|---|
| Initial researcher pool | ≈100 nuclear physicists | ≈1,500 safety researchers | ≈1,500 |
| Peak workforce | ≈6,000 scientists/engineers | ≈2,000 (projected 2027) | ≈10,000 (potential) |
| Scaling factor | 60x in 3 years | 1.3x in 3 years | 6.7x in 3 years |
| Government priority | Maximum | Minimal | Hypothetical high |
| Resource allocation | $28B (2020 dollars) | ≈$500M annually | $5-10B annually |
Other Technology Mobilizations
| Program | Duration | Talent Scale-up | Success Factors |
|---|---|---|---|
| Apollo Program | 8 years | 20x | Clear goal, unlimited resources |
| COVID vaccine development | 1 year | 5x | Existing infrastructure, parallel efforts |
| Cold War cryptography | 10 years | 15x | Security priority, university partnerships |
Feedback Loop Analysis
Positive Feedback Loops
Research Quality → Field Attraction:
- High-impact safety research increases field prestige
- Prestigious field attracts top-tier researchers
- Better researchers produce higher-impact research
Success → Funding → Scale:
- Visible safety progress builds funder confidence
- Increased funding enables program expansion
- Larger programs achieve economies of scale
Negative Feedback Loops
Capability Race → Brain Drain:
- AI race intensifies, driving higher capability salaries
- Safety researchers transition to better-compensated roles
- Reduced safety talent further slows progress
Progress Pessimism → Attrition:
- Slow safety progress relative to capabilities
- Researcher demoralization and career changes
- Talent loss further slows progress
Geographic Distribution
Current Concentration
| Region | Safety Researchers | Major Organizations | Constraints |
|---|---|---|---|
| SF Bay Area | 40-50% | Anthropic, OpenAI, MIRI | High cost of living |
| Boston/Cambridge | 15-20% | MIT, Harvard | Limited industry positions |
| London | 10-15% | DeepMind, Oxford | Visa requirements |
| Other US | 15-20% | Various universities | Geographic dispersion |
| Other International | 10-15% | Scattered | Visa, funding constraints |
Geographic Bottlenecks
Visa and Immigration Issues:
- H-1B lottery system blocks international talent
- Security clearance requirements limit government roles
- Brexit complications affect EU-UK movement
Regional Capacity Constraints:
- Housing costs in AI hubs (SF, Boston) limit accessibility
- Limited remote work policies at some organizations
- Talent concentration reduces geographic resilience
Quality vs. Quantity Trade-offs
Researcher Tier Analysis
| Tier | Characteristics | Current Supply | Needed Supply | Impact Multiple |
|---|---|---|---|---|
| A-tier | Can lead research agendas, mentor others | 50-100 | 200-400 | 10-50x average |
| B-tier | Independent research, implementation | 200-500 | 800-1,200 | 3-5x average |
| C-tier | Execution, support roles | 500-1,000 | 1,000-2,000 | 1x baseline |
| D-tier | Adjacent skills, potential | 1,000+ | Variable | 0.3-0.5x |
Strategic Implications
Leadership Bottleneck: The shortage of A-tier researchers who can set research directions and mentor others may be more critical than total headcount.
Optimal Resource Allocation:
- High-leverage: Develop A-tier researchers (long-term, high-cost)
- Medium-leverage: Scale B-tier production (medium-term, medium-cost)
- Low-leverage: Increase C-tier volume (short-term, low-cost)
Economic Impact Analysis
Opportunity Cost Assessment
The talent shortage imposes significant opportunity costs on AI safety progress:
| Lost Progress Type | Annual Value | Cumulative Impact |
|---|---|---|
| Research breakthroughs delayed | $100-500M | Compound delay in safety solutions |
| Interpretability progress | $50-200M | Reduced understanding of systems |
| Governance preparation | $20-100M | Policy lag behind technology |
| Total opportunity cost | $170-800M/year | Exponential safety lag |
Return on Investment
Talent development interventions show strong ROI compared to opportunity costs:
| Investment | Annual Cost | Researchers Added | ROI (5-year) |
|---|---|---|---|
| Training programs | $100M | 500 | 5-10x |
| Retention programs | $100M | 200 (net) | 3-7x |
| Infrastructure | $50M | 100 | 4-8x |
| Combined program | $250M | 800 | 4-9x |
Policy Recommendations
Immediate Actions (2025)
-
Scale Proven Programs:
- Triple funding for MATS-style programs ($45M investment)
- Expand ARENA↗🔗 webARENA – AI Safety EducationARENA is a widely recommended starting point for technically skilled individuals looking to enter AI safety research, and its curriculum materials are frequently referenced in AI safety onboarding guides and reading lists.ARENA (Alignment Research Engineer Accelerator) is an educational program designed to train technical AI safety researchers by providing structured curriculum covering mechanist...ai-safetyalignmentinterpretabilitytechnical-safety+5Source ↗ and similar bootcamps
- Create industry-to-safety transition scholarships
-
Remove Friction:
- Streamline H-1B process for AI safety roles
- Create safety-specific grant categories
- Establish talent-sharing agreements between organizations
Medium-term Reforms (2025-2027)
-
Institutional Development:
- Fund 10-20 new AI safety PhD programs
- Establish government AI safety research fellowships
- Create safety-focused postdoc exchange programs
-
Competitive Balance:
- Safety researcher salary competitiveness fund
- Equity/ownership programs at safety organizations
- Long-term career advancement pathways
Long-term Infrastructure (2027-2030)
-
National Capacity Building:
- AI Safety Corps (government service program)
- National AI Safety University Consortium
- International talent exchange agreements
-
Systemic Changes:
- Safety research requirements for AI development
- Academic tenure track positions in safety
- Industry safety certification programs
Key Uncertainties and Cruxes
Key Questions
- ?How much additional research progress would each marginal safety researcher actually produce?
- ?Can training time be compressed from years to months without quality loss?
- ?Will competition from capabilities research permanently prevent salary competitiveness?
- ?What fraction of the 'adjacent' researcher pool could realistically transition to safety focus?
- ?How much does geographic distribution matter for research productivity and coordination?
- ?What is the optimal ratio between A-tier, B-tier, and C-tier researchers?
Critical Research Questions
- Marginal Impact Assessment: Quantifying the relationship between researcher quantity/quality and safety progress
- Training Optimization: Identifying minimum viable training for productive safety research
- Retention Psychology: Understanding what motivates long-term commitment to safety work
- Coordination Effects: Measuring productivity gains from researcher collaboration and proximity
Model Limitations and Biases
Data Quality Issues
- Definition Ambiguity: No consensus on what constitutes "AI safety research"
- Hidden Supply: Many researchers work on safety-relevant problems without identifying as safety researchers
- Quality Assessment: Subjective researcher quality ratings introduce bias
- Rapid Change: Field dynamics evolve faster than data collection cycles
Methodological Limitations
- Linear Assumptions: Model assumes linear relationships between resources and outcomes
- Quality-Quantity Simplification: Real productivity relationships are complex and nonlinear
- Geographic Aggregation: Treats globally distributed talent as fungible
- Temporal Lag Ignoring: Training and productivity gaps have complex timing relationships
Prediction Uncertainties
- Scenario Dependence: Projections highly sensitive to AI development trajectory
- Policy Response: Unknown government/industry response to demonstrated AI risks
- Technology Disruption: New training methods or research tools could change dynamics
- Field Evolution: Safety research priorities and methods continue evolving
Related Risk Models
This talent gap model connects to several other risks that could compound or mitigate the shortage:
- Expertise Atrophy: If AI tools replace human expertise, safety researcher skills may degrade
- Racing Dynamics: Competition between labs drives talent toward capabilities rather than safety
- Flash Dynamics: Rapid AI development could outpace even scaled talent pipelines
- Scientific Corruption: Poor incentives could reduce effective research output per researcher
Strategic Implications
The talent shortage represents a foundational constraint on AI safety progress that could determine whether adequate safety research occurs before advanced AI deployment. Unlike funding or technical challenges, talent development has long lead times that make delays especially costly.
For Organizations: Talent competition will likely intensify, making retention strategies and alternative talent sources critical for organizational success.
For Policymakers: Early intervention in talent development could provide significant leverage over long-term AI safety outcomes, while delayed action may prove ineffective.
For Individual Researchers: Career decisions made in the next 2-3 years could have outsized impact on field development during a critical period.
Sources and Resources
Research and Analysis
| Source | Type | Key Findings |
|---|---|---|
| 80,000 Hours AI Safety Career Reviews↗🔗 web★★★☆☆80,000 HoursAI safety technical research | Career review | 80,000 HoursA practical career-guidance resource from 80,000 Hours aimed at individuals considering entering AI safety research; useful for understanding field entry points and skill requirements rather than technical research content itself.80,000 Hours provides a comprehensive career guide for technical AI safety research, covering empirical and theoretical paths, entry requirements, key organizations, and strateg...ai-safetyalignmenttechnical-safetyexistential-risk+4Source ↗ | Career analysis | Talent bottlenecks, career pathways |
| Coefficient Giving AI Grant Database↗🔗 web★★★★☆Coefficient GivingOpen Philanthropy AI Grant DatabaseThis resource is mislabeled; it is Coefficient Giving's fund homepage, not Open Philanthropy's grant database. It lists a 'Navigating Transformative AI' fund but provides no grant-level detail relevant to AI safety research tracking.This page appears to be Coefficient Giving's fund listing, not the Open Philanthropy AI grant database as labeled. It describes a philanthropic fund called 'Navigating Transform...ai-safetygovernanceexistential-riskcoordination+2Source ↗ | Funding data | Investment patterns, organization capacity |
| MATS Program Outcomes↗🔗 webMATS Research ProgramMATS is one of the primary talent pipelines into the AI safety field; wiki users interested in career transitions or field-building efforts should consider this a key institutional reference.MATS is an intensive fellowship program designed to help researchers transition into AI safety careers, offering structured mentorship from leading researchers, stipends, and co...ai-safetyalignmentfield-buildingeducational+4Source ↗ | Training data | Completion rates, placement success |
| AI Safety Support Talent Survey↗🔗 webAI Safety Support Talent SurveyUseful for individuals looking to enter AI safety careers or organizations seeking to understand workforce trends; complements technical AI safety research by addressing human capital and pipeline challenges.AI Safety Support is an organization focused on growing and supporting the AI safety talent pipeline by helping researchers, engineers, and other professionals enter and thrive ...ai-safetyfield-buildingcoordinationgovernance+2Source ↗ | Field survey | Researcher demographics, career paths |
Training Programs and Organizations
| Program | Focus | Contact |
|---|---|---|
| MATS (ML Alignment & Theory Scholars)↗🔗 webMATS Research ProgramMATS is one of the primary talent pipelines into the AI safety field; wiki users interested in career transitions or field-building efforts should consider this a key institutional reference.MATS is an intensive fellowship program designed to help researchers transition into AI safety careers, offering structured mentorship from leading researchers, stipends, and co...ai-safetyalignmentfield-buildingeducational+4Source ↗ | Research training | applications@matsprogram.org |
| ARENA (AI Research Extensive Alliance)↗🔗 webARENA – AI Safety EducationARENA is a widely recommended starting point for technically skilled individuals looking to enter AI safety research, and its curriculum materials are frequently referenced in AI safety onboarding guides and reading lists.ARENA (Alignment Research Engineer Accelerator) is an educational program designed to train technical AI safety researchers by providing structured curriculum covering mechanist...ai-safetyalignmentinterpretabilitytechnical-safety+5Source ↗ | Technical bootcamps | contact@arena.education |
| AI Safety Support↗🔗 webAI Safety Support Talent SurveyUseful for individuals looking to enter AI safety careers or organizations seeking to understand workforce trends; complements technical AI safety research by addressing human capital and pipeline challenges.AI Safety Support is an organization focused on growing and supporting the AI safety talent pipeline by helping researchers, engineers, and other professionals enter and thrive ...ai-safetyfield-buildingcoordinationgovernance+2Source ↗ | Career guidance | advice@aisafetysupport.org |
| 80,000 Hours↗🔗 web★★★☆☆80,000 Hours80,000 Hours methodology80,000 Hours is a major talent and career funnel into the AI safety ecosystem; useful for understanding how researchers and practitioners are recruited into the field and what career paths are considered high-impact by the effective altruism community.80,000 Hours is a nonprofit that provides research and advice on how to use your career to have the most positive impact on the world's most pressing problems, with significant ...ai-safetyexistential-riskgovernancefield-building+3Source ↗ | Career planning | team@80000hours.org |
Policy and Governance Resources
| Organization | Focus | Link |
|---|---|---|
| Centre for AI Governance↗🏛️ government★★★★☆Centre for the Governance of AIGovAI helps decision-makers navigate the transition to a world with advanced AI, by producing rigorous research and fostering talent." name="description"/><meta content="GovAI | HomeGovAI is one of the most prominent AI governance research organizations globally; their publications on AI policy, international coordination, and existential risk governance are frequently cited in AI safety literature and policy discussions.The Centre for the Governance of AI (GovAI) is a leading research organization dedicated to helping decision-makers navigate the transition to a world with advanced AI. It produ...governanceai-safetypolicyexistential-risk+4Source ↗ | Policy research | https://www.governance.ai/ |
| Partnership on AI↗🔗 web★★★☆☆Partnership on AIPartnership on AI (PAI) – Multi-Stakeholder AI Governance OrganizationPAI is a major multi-stakeholder governance body relevant to AI safety researchers interested in policy coordination, industry norms, and the institutional landscape surrounding responsible AI deployment.Partnership on AI (PAI) is a nonprofit coalition of AI researchers, civil society organizations, academics, and companies working to develop best practices, conduct research, an...governanceai-safetypolicycoordination+2Source ↗ | Industry coordination | https://www.partnershiponai.org/ |
| Future of Humanity Institute↗🔗 web★★★★☆Future of Humanity Institute**Future of Humanity Institute**FHI was a pioneering institution in AI safety and existential risk; this archived homepage is useful for historical context and understanding the institutional origins of the field, though the site is no longer actively updated following its April 2024 closure.The official website of the Future of Humanity Institute (FHI), an Oxford University research center that was foundational in establishing the fields of existential risk researc...ai-safetyexistential-riskalignmentgovernance+3Source ↗ | Long-term research | https://web.archive.org/web/2024/https://www.fhi.ox.ac.uk/ |
References
Partnership on AI (PAI) is a nonprofit coalition of AI researchers, civil society organizations, academics, and companies working to develop best practices, conduct research, and shape policy around responsible AI development. It brings together diverse stakeholders to address challenges including safety, fairness, transparency, and the societal impacts of AI systems. PAI serves as a coordination hub for cross-sector dialogue on AI governance.
The official website of the Future of Humanity Institute (FHI), an Oxford University research center that was foundational in establishing the fields of existential risk research and AI safety. FHI closed on 16 April 2024 after approximately two decades of influential work. The site now serves as an archived record of the institution's history, research agenda, and legacy.
This page appears to be Coefficient Giving's fund listing, not the Open Philanthropy AI grant database as labeled. It describes a philanthropic fund called 'Navigating Transformative AI' focused on ensuring AI is safe and well-governed, alongside other cause areas evaluated using importance, neglectedness, and tractability criteria.
Anthropic's careers page outlines the company's mission to build safe and beneficial AI, highlighting their guiding principles and inviting researchers, engineers, and builders to join their work on Claude and AI safety. The page emphasizes a 'race to the top' on safety culture and a commitment to ensuring powerful AI benefits humanity.
80,000 Hours provides a comprehensive career guide for technical AI safety research, covering empirical and theoretical paths, entry requirements, key organizations, and strategies for building relevant skills. It frames AI safety research as one of the highest-impact career choices given existential risks from advanced AI systems.
AI Safety Support is an organization focused on growing and supporting the AI safety talent pipeline by helping researchers, engineers, and other professionals enter and thrive in the field. It provides career guidance, mentorship matching, and conducts surveys to understand supply and demand dynamics within the AI safety workforce. The initiative aims to reduce bottlenecks in field-building by connecting aspiring AI safety professionals with opportunities and resources.
ARENA (Alignment Research Engineer Accelerator) is an educational program designed to train technical AI safety researchers by providing structured curriculum covering mechanistic interpretability, reinforcement learning, and other core alignment topics. It aims to build the pipeline of competent AI safety engineers by offering hands-on, project-based learning. The program serves as a key field-building initiative to address the talent gap in technical AI safety research.
An overview of the Manhattan Project, the large-scale US government program during WWII that developed the first nuclear weapons. It serves as a historical case study of rapid, secretive, high-stakes scientific mobilization with profound and lasting global consequences. This resource is relevant to AI safety discussions as an analogy for transformative technology development under urgency and secrecy.
MATS is an intensive fellowship program designed to help researchers transition into AI safety careers, offering structured mentorship from leading researchers, stipends, and community integration. Since 2021, it has trained over 446 researchers who have collectively produced 150+ research papers and gone on to work at top AI safety organizations.
Open Philanthropy is a major philanthropic organization that funds work across global health, AI safety, biosecurity, and other cause areas. Their grants database provides transparency into which organizations and research directions receive funding. They are one of the largest funders of AI safety and existential risk research.
The OpenAI careers page lists open positions across research, engineering, policy, and operations roles at OpenAI. It serves as a recruiting hub for one of the leading AI labs, reflecting the organization's hiring priorities and team structure. Relevant to understanding where AI safety and capabilities talent is being directed.
80,000 Hours is a nonprofit that provides research and advice on how to use your career to have the most positive impact on the world's most pressing problems, with significant focus on AI safety and existential risk. They offer career guides, job boards, and in-depth research on high-priority cause areas and career paths. Their methodology emphasizes earning to give, direct work in high-impact fields, and building career capital.
The Centre for the Governance of AI (GovAI) is a leading research organization dedicated to helping decision-makers navigate the transition to a world with advanced AI. It produces rigorous research on AI governance, policy, and societal impacts, while fostering a global talent pipeline for responsible AI oversight. GovAI bridges technical AI safety concerns with practical policy recommendations.