Page StatusContent
Edited 2 weeks ago1.7k words1 backlinks
40
QualityAdequateQuality: 40/100Human-assigned rating of overall page quality, considering depth, accuracy, and completeness.Structure suggests 6725
ImportancePeripheralImportance: 25/100How central this topic is to AI safety. Higher scores mean greater relevance to understanding or mitigating AI risk.10
Structure10/15Structure: 10/15Automated score based on measurable content features.Word count2/2Tables3/3Diagrams0/2Internal links2/2Citations0/3Prose ratio1/2Overview section1/114TablesData tables in the page0DiagramsCharts and visual diagrams53Internal LinksLinks to other wiki pages0FootnotesFootnote citations [^N] with sources0External LinksMarkdown links to outside URLs%30%Bullet RatioPercentage of content in bullet lists
Updated every 6 weeksDue in 4 weeks
Summary
Holden Karnofsky directed $300M+ in AI safety funding through Open Philanthropy, growing the field from ~20 to 400+ FTE researchers and developing influential frameworks like the 'Most Important Century' thesis (15% transformative AI by 2036, 50% by 2060). His funding decisions include a $580M Anthropic investment and establishment of 15+ university AI safety programs.
Issues1
QualityRated 40 but structure suggests 67 (underrated by 27 points)
Holden Karnofsky
Person
Holden Karnofsky
Holden Karnofsky directed $300M+ in AI safety funding through Open Philanthropy, growing the field from ~20 to 400+ FTE researchers and developing influential frameworks like the 'Most Important Century' thesis (15% transformative AI by 2036, 50% by 2060). His funding decisions include a $580M Anthropic investment and establishment of 15+ university AI safety programs.
AffiliationCoefficient Giving
RoleCo-CEO
Known ForDirecting billions toward AI safety, effective altruism leadership, AI timelines work
Websiteopenphilanthropy.org
Related
Organizations
AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding...
People
Toby OrdPersonToby OrdComprehensive biographical profile of Toby Ord documenting his 10% AI extinction estimate and role founding effective altruism, with detailed tables on risk assessments, academic background, and in...Quality: 41/100
1.7k words · 1 backlinks
Person
Holden Karnofsky
Holden Karnofsky directed $300M+ in AI safety funding through Open Philanthropy, growing the field from ~20 to 400+ FTE researchers and developing influential frameworks like the 'Most Important Century' thesis (15% transformative AI by 2036, 50% by 2060). His funding decisions include a $580M Anthropic investment and establishment of 15+ university AI safety programs.
AffiliationCoefficient Giving
RoleCo-CEO
Known ForDirecting billions toward AI safety, effective altruism leadership, AI timelines work
Websiteopenphilanthropy.org
Related
Organizations
AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding...
People
Toby OrdPersonToby OrdComprehensive biographical profile of Toby Ord documenting his 10% AI extinction estimate and role founding effective altruism, with detailed tables on risk assessments, academic background, and in...Quality: 41/100
1.7k words · 1 backlinks
Overview
Holden Karnofsky was co-CEO of Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100↗🔗 webOpen Philanthropy grants databaseOpen Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, an...x-riskresource-allocationresearch-prioritiesoptimization+1Source ↗ (formerly Coefficient GivingOrganizationOpen PhilanthropyOpen Philanthropy rebranded to Coefficient Giving in November 2025. See the Coefficient Giving page for current information.), the most influential grantmaker in AI safety and existential risk. Through Coefficient, he directed over $100 million toward AI safety research and governance, fundamentally transforming it from a fringe academic interest into a well-funded field with hundreds of researchers. In 2025, he joined Anthropic.
His strategic thinking has shaped how the effective altruism community prioritizes AI risk through frameworks like the "Most Important Century"↗🔗 web"Most Important Century"effective-altruismai-safety-fundingai-timelinesSource ↗ thesis. This argues we may live in the century that determines humanity's entire future trajectory due to transformative AI development.
| Funding Achievement | Amount | Impact |
|---|---|---|
| Total AI safety grants | $300M+ | Enabled field growth from ~dozens to hundreds of researchers |
| Anthropic investment | $580M+ | Created major safety-focused AI lab |
| Field building grants | $50M+ | Established academic programs and research infrastructure |
Risk Assessment
| Risk Category | Karnofsky's Assessment | Evidence | Timeline |
|---|---|---|---|
| Transformative AI | ~15% by 2036, ≈50% by 2060 | Bio anchors framework↗🔗 webBio anchors frameworkeffective-altruismai-safety-fundingai-timelinesSource ↗ | This century |
| Existential importance | "Most important century" | AI could permanently shape humanity's trajectory | 2021-2100 |
| Tractability | High enough for top priority | Open Phil's largest focus area allocation | Current |
| Funding adequacy | Severely underfunded | Still seeking to grow field substantially | Ongoing |
Career Evolution and Major Achievements
Early Career (2007-2014): Building Effective Altruism
| Period | Role | Key Achievements |
|---|---|---|
| 2007-2011 | Co-founder, GiveWell↗🔗 webGiveWellcost-effectivenessresearch-prioritiesexpected-valueeffective-altruism+1Source ↗ | Pioneered rigorous charity evaluation methodology |
| 2011-2014 | Launch Coefficient Giving | Expanded beyond global health to cause prioritization |
| 2012-2014 | EA movement building | Helped establish effective altruism as global movement |
Transition to AI Focus (2014-2018)
Initial AI engagement:
- 2014: First significant AI safety grants through Coefficient (then Open Philanthropy)
- 2016: Major funding to Center for Human-Compatible AI (CHAI)OrganizationCenter for Human-Compatible AICHAI is UC Berkeley's AI safety research center founded by Stuart Russell in 2016, pioneering cooperative inverse reinforcement learning and human-compatible AI frameworks. The center has trained 3...Quality: 37/100
- 2017: Early OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to commercial AGI developer, with detailed analysis of governance crisis, safety researcher exodus (75% of ... funding (before pivot to for-profit)
- 2018: Increased conviction leading to AI as top priority
AI Safety Leadership (2018-Present)
Major funding decisions:
- 2021: $580M investment in Anthropic↗🔗 web★★★★☆Anthropic$580M investment in Anthropiceffective-altruismai-safety-fundingai-timelinesSource ↗ to create safety-focused lab
- 2022: Establishment of AI safety university programs↗🔗 webAI safety university programssafetyeffective-altruismai-safety-fundingai-timelinesSource ↗
- 2023: Expanded governance funding addressing AI regulation
Strategic Frameworks and Intellectual Contributions
The "Most Important Century" Thesis
Core argument structure:
| Component | Claim | Implication |
|---|---|---|
| Technology potential | Transformative AI possible this century | Could exceed agricultural/industrial revolution impacts |
| Speed differential | AI transition faster than historical precedents | Less time to adapt and coordinate |
| Leverage moment | Our actions now shape outcomes | Unlike past revolutions where individuals had little influence |
| Conclusion | This century uniquely important | Justifies enormous current investment |
Supporting evidence:
- Biological anchors methodology↗🔗 webBio anchors frameworkeffective-altruismai-safety-fundingai-timelinesSource ↗ for AI timelines
- Historical analysis of technological transitions
- Economic modeling of AI impact potential
Bio Anchors Framework
Developed with Ajeya Cotra↗🔗 webAjeya Cotraeffective-altruismai-safety-fundingai-timelinesSource ↗, this framework estimates AI development timelines by comparing required computation to biological systems:
| Anchor Type | Computation Estimate | Timeline Implication |
|---|---|---|
| Human brain | ≈10^15 FLOP/s | Medium-term (2030s-2040s) |
| Human lifetime | ≈10^24 FLOP | Longer-term (2040s-2050s) |
| Evolution | ≈10^41 FLOP | Much longer-term if needed |
Coefficient Giving Funding Strategy
Portfolio Approach
| Research Area | Funding Focus | Key Recipients | Rationale |
|---|---|---|---|
| Technical alignment | $100M+ | AnthropicOrganizationAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding..., Redwood ResearchOrganizationRedwood ResearchA nonprofit AI safety and security research organization founded in 2021, known for pioneering AI Control research, developing causal scrubbing interpretability methods, and conducting landmark ali...Quality: 78/100 | Direct work on making AI systems safer |
| AI governanceParameterAI GovernanceThis page contains only component imports with no actual content - it displays dynamically loaded data from an external source that cannot be evaluated. | $80M+ | Center for Security and Emerging TechnologyOrganizationCSET (Center for Security and Emerging Technology)CSET is a $100M+ Georgetown center with 50+ staff conducting data-driven AI policy research, particularly on U.S.-China competition and export controls. The center conducts hundreds of annual gover...Quality: 43/100↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsI apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. ...prioritizationresource-allocationportfolioescalation+1Source ↗, policy fellowships | Institutional responses to AI development |
| Field building | $50M+ | University programs, individual researchers | Growing research community |
| Compute governance | $20M+ | Compute monitoring researchPolicyCompute MonitoringAnalyzes two compute monitoring approaches: cloud KYC (implementable in 1-2 years, covers ~60% of frontier training via AWS/Azure/Google) and hardware governance (3-5 year timeline). Cloud KYC targ...Quality: 69/100 | Oversight of AI development resources |
Grantmaking Philosophy
Key principles:
- Hits-based giving: Expect most grants to have limited impact, few to be transformative
- Long time horizons: Patient capital for 5-10 year research projects
- Active partnership: Strategic guidance beyond just funding
- Portfolio diversification: Multiple approaches given uncertainty
Notable funding decisions:
- Anthropic investment↗🔗 web★★★★☆Anthropic$580M investment in Anthropiceffective-altruismai-safety-fundingai-timelinesSource ↗: $580M to create safety-focused competitor to OpenAI
- MIRI fundingOrganizationMachine Intelligence Research InstituteComprehensive organizational history documenting MIRI's trajectory from pioneering AI safety research (2000-2020) to policy advocacy after acknowledging research failure, with detailed financial da...Quality: 50/100: Early support for foundational AI alignmentApproachAI AlignmentComprehensive review of AI alignment approaches finding current methods (RLHF, Constitutional AI) achieve 75-90% effectiveness on existing systems but face critical scalability challenges, with ove...Quality: 91/100 research
- Policy fellowships: Placing AI safety researchers in government positions
Current Views and Assessment
Karnofsky's AI Risk Timeline
Based on public statements and Coefficient Giving priorities from 2023-2024, Karnofsky's views reflect a combination of timeline estimates derived from technical forecasting and strategic assessments about field readiness and policy urgency:
| Expert/Source | Estimate | Reasoning |
|---|---|---|
| Transformative AI (2022) | 15% by 2036, 50% by 2060 | Derived from the bio anchors framework developed with Ajeya Cotra, which estimates AI development timelines by comparing required computation to biological systems. This central estimate suggests transformative AI is more likely than not within this century, though substantial uncertainty remains around both shorter and longer timelines. |
| Field adequacy (2024) | Still severely underfunded | Despite directing over $100M toward AI safety and growing the field from approximately 20 to 400+ FTE researchers, Coefficient Giving continues aggressive hiring and grantmaking. This assessment reflects the belief that the scale of the challenge—ensuring safe development of transformative AI—far exceeds current resources and talent devoted to it. |
| Policy urgency (2024) | High priority | Coefficient has significantly increased governance focus, funding policy research, placing fellows in government positions, and supporting regulatory frameworks. This shift recognizes that technical alignment work alone is insufficient—institutional and policy responses are critical to managing AI development trajectories and preventing racing dynamicsRiskAI Development Racing DynamicsRacing dynamics analysis shows competitive pressure has shortened safety evaluation timelines by 40-60% since ChatGPT's launch, with commercial labs reducing safety work from 12 weeks to 4-6 weeks....Quality: 72/100. |
Evolution of Views (2020-2024)
| Year | Key Update | Reasoning |
|---|---|---|
| 2021 | "Most Important Century" series | Crystallized long-term strategic thinking |
| 2022 | Increased policy focus | Recognition of need for governance alongside technical work |
| 2023 | Anthropic model success | Validation of safety-focused lab approach |
| 2024 | Accelerated timelines concern | Shorter timelines than bio anchors suggested↗🔗 webShorter timelines than bio anchors suggestedeffective-altruismai-safety-fundingai-timelinesSource ↗ |
Influence on AI Safety Field
Field Growth Metrics
| Metric | 2015 | 2024 | Growth Factor |
|---|---|---|---|
| FTE researchers | ≈20 | ≈400 | 20x |
| Annual funding | <$5M | >$200M | 40x |
| University programs | 0 | 15+ | New category |
| Major organizations | 2-3 | 20+ | 7x |
Institutional Impact
Academic legitimacy:
- Funding enabled AI safety courses↗🔗 webAI safety coursessafetyeffective-altruismai-safety-fundingai-timelinesSource ↗ at major universities
- Supported tenure-track positions focused on alignment research
- Created pathway for traditional CS researchers to enter field
Policy influence:
- Funded experts now advising US AI Safety InstituteOrganizationUS AI Safety InstituteThe US AI Safety Institute (AISI), established November 2023 within NIST with $10M budget (FY2025 request $82.7M), conducted pre-deployment evaluations of frontier models through MOUs with OpenAI a...Quality: 91/100
- Supported research informing EU AI ActPolicyEU AI ActComprehensive overview of the EU AI Act's risk-based regulatory framework, particularly its two-tier approach to foundation models that distinguishes between standard and systemic risk AI systems. ...Quality: 55/100↗🔗 webEU AI ActThe EU AI Act introduces the world's first comprehensive AI regulation, classifying AI applications into risk categories and establishing legal frameworks for AI development and...governancesoftware-engineeringcode-generationprogramming-ai+1Source ↗
- Built relationships between AI safety community and policymakers
Key Uncertainties and Strategic Cruxes
Open Questions in Karnofsky's Framework
| Uncertainty | Stakes | Current Evidence |
|---|---|---|
| AI timeline accuracy | Entire strategy timing | Mixed signals from recent capabilities |
| Technical tractability | Funding allocation efficiency | Early positive results but limited validation |
| Governance effectiveness | Policy investment value | Unclear institutional responsiveness |
| Anthropic success | Large investment justification | Strong early results but long-term unknown |
Strategic Disagreements
Within EA community:
- Some argue for longtermist focus beyond AIConceptLong-Timelines Technical WorldviewComprehensive overview of the long-timelines worldview (20-40+ years to AGI, 5-20% P(doom)), arguing for foundational research over rushed solutions based on historical AI overoptimism, current sys...Quality: 91/100
- Others prefer global health and development↗🔗 webGiveWellcost-effectivenessresearch-prioritiesexpected-valueeffective-altruism+1Source ↗ emphasis
- Debate over concentration vs. diversification of funding
With AI safety researchers:
- Tension between technical alignment focusArgumentWhy Alignment Might Be HardComprehensive synthesis of why AI alignment is fundamentally difficult, covering specification problems (value complexity, Goodhart's Law), inner alignment failures (mesa-optimization, deceptive al...Quality: 61/100 and governance approaches
- Disagreement over open vs. closed developmentCruxOpen vs Closed Source AIComprehensive analysis of open vs closed source AI debate, documenting that open model performance gap narrowed from 8% to 1.7% in 2024, with 1.2B+ Llama downloads by April 2025 and DeepSeek R1 dem...Quality: 60/100 funding
- Questions about emphasis on capabilities research safety benefits
Public Communication and Influence
Cold Takes Blog Impact
Most influential posts:
- "The Most Important Century"↗🔗 web"Most Important Century"effective-altruismai-safety-fundingai-timelinesSource ↗ series (>100k views)
- "AI Timelines: Where the Arguments Stand"↗🔗 webShorter timelines than bio anchors suggestedeffective-altruismai-safety-fundingai-timelinesSource ↗ (policy reference)
- "Bio Anchors" explanation↗🔗 webBio anchors frameworkeffective-altruismai-safety-fundingai-timelinesSource ↗ (research methodology)
Communication approach:
- Transparent reasoning and uncertainty acknowledgment
- Accessible explanations of complex topics
- Regular updates as views evolve
- Direct engagement with critics and alternative viewpoints
Media and Policy Engagement
| Platform | Reach | Impact |
|---|---|---|
| Congressional testimony | Direct policy influence | Informed AI regulation debateCruxGovernment Regulation vs Industry Self-GovernanceComprehensive comparison of government regulation versus industry self-governance for AI, documenting that US federal AI regulations doubled to 59 in 2024 while industry lobbying surged 141% to 648...Quality: 54/100 |
| Academic conferences | Research community | Shaped university AI safety programs |
| EA GlobalOrganizationEA GlobalEA Global is a series of selective conferences organized by the Centre for Effective Altruism that connects committed EA practitioners to collaborate on global challenges, with AI safety becoming i...Quality: 38/100 talks | Movement direction | Influenced thousands of career decisions |
| Podcast interviews | Public understanding | Mainstream exposure for AI safety ideas |
Current Priorities and Future Direction
2024-2026 Strategic Focus
Immediate priorities:
- Anthropic scaling: Supporting responsible development of powerful systems
- Governance acceleration: Policy research and implementation support
- Technical diversification: Funding multiple alignment research approaches
- International coordinationAi Transition Model ParameterInternational CoordinationThis page contains only a React component placeholder with no actual content rendered. Cannot assess importance or quality without substantive text.: Supporting global AI safety cooperation
Emerging areas:
- Compute governance infrastructure
- AI evaluationApproachAI EvaluationComprehensive overview of AI evaluation methods spanning dangerous capability assessment, safety properties, and deception detection, with categorized frameworks from industry (Anthropic Constituti...Quality: 72/100 methodologies
- Corporate AI safetyApproachCorporate AI Safety ResponsesMajor AI labs invest $300-500M annually in safety (5-10% of R&D) through responsible scaling policies and dedicated teams, but face 30-40% safety team turnover and significant implementation gaps b...Quality: 68/100 practices
- Prediction marketApproachPrediction Markets (AI Forecasting)Prediction markets achieve Brier scores of 0.16-0.24 (15-25% better than polls) by aggregating dispersed information through financial incentives, with platforms handling $1-3B annually. For AI saf...Quality: 56/100 applications
Long-term Vision
Field development goals:
- Self-sustaining research ecosystem independent of Coefficient Giving
- Government funding matching or exceeding philanthropic support
- Integration of safety research into mainstream AI development
- International coordination mechanismsPolicyInternational Coordination MechanismsComprehensive analysis of international AI coordination mechanisms shows growing but limited progress: 11-country AI Safety Institute network with ~$200M budget expanding to include India; Council ...Quality: 91/100 for AI governance
Critiques and Responses
Common Criticisms
| Criticism | Karnofsky's Response | Counter-evidence |
|---|---|---|
| Over-concentration of powerRiskAI-Driven Concentration of PowerDocuments how AI development is concentrating in ~20 organizations due to $100M+ compute costs, with 5 firms controlling 80%+ of cloud infrastructure and projections reaching $1-10B per model by 20...Quality: 65/100 | Funding diversification, transparency | Multiple other major funders emerging |
| Field capture risk | Portfolio approach, external evaluation | Continued criticism tolerated and addressed |
| Timeline overconfidence | Explicit uncertainty, range estimates | Regular updating based on new evidence |
| Governance skepticism | Measured expectations, multiple approaches | Early policy wins demonstrate tractability |
Ongoing Debates
Resource allocation:
- Should Coefficient Giving fund more basic research vs. applied safety work?
- Optimal balance between technical and governance approaches?
- Geographic distribution of funding (US-centric concerns)
Strategic approach:
- Speed vs. care in scaling funding
- Competition vs. cooperation with AI labs
- Public advocacy vs. behind-the-scenes influence
Sources & Resources
Primary Sources
| Type | Source | Description |
|---|---|---|
| Blog | Cold Takes↗🔗 webCold Takeseffective-altruismai-safety-fundingai-timelinesSource ↗ | Karnofsky's strategic thinking and analysis |
| Organization | Coefficient Giving↗🔗 webOpen Philanthropy grants databaseOpen Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, an...x-riskresource-allocationresearch-prioritiesoptimization+1Source ↗ | Grant database and reasoning |
| Research | Bio Anchors Report↗🔗 webBio Anchors Reporteffective-altruismai-safety-fundingai-timelinesSource ↗ | Technical forecasting methodology |
| Testimony | Congressional Hearing↗🏛️ government★★★★★US CongressCongressional Hearingeffective-altruismai-safety-fundingai-timelinesSource ↗ | Policy positions and recommendations |
Secondary Analysis
| Type | Source | Focus |
|---|---|---|
| Academic | EA Research↗✏️ blog★★★☆☆EA ForumEA Forum Career Poststalentfield-buildingcareer-transitionsprioritization+1Source ↗ | Critical analysis of funding decisions |
| Journalistic | MIT Technology Review↗🔗 web★★★★☆MIT Technology ReviewMIT Technology Review: Deepfake Coverageai-forecastingcompute-trendstraining-datasetsconstitutional-ai+1Source ↗ | External perspective on influence |
| Policy | RAND Corporation↗🔗 web★★★★☆RAND CorporationRANDRAND conducts policy research analyzing AI's societal impacts, including potential psychological and national security risks. Their work focuses on understanding AI's complex im...governancecybersecurityprioritizationresource-allocation+1Source ↗ | Government research on philanthropic AI funding |
Related Profiles
- Dario AmodeiPersonDario AmodeiComprehensive biographical profile of Anthropic CEO Dario Amodei documenting his 'race to the top' philosophy, 10-25% catastrophic risk estimate, 2026-2030 AGI timeline, and Constitutional AI appro...Quality: 41/100 - CEO of Anthropic, major funding recipient
- Paul ChristianoPersonPaul ChristianoComprehensive biography of Paul Christiano documenting his technical contributions (IDA, debate, scalable oversight), risk assessment (~10-20% P(doom), AGI 2030s-2040s), and evolution from higher o...Quality: 39/100 - Technical alignment researcher, influenced Karnofsky's views
- Nick BostromPersonNick BostromComprehensive biographical profile of Nick Bostrom covering his founding of FHI, the landmark 2014 book 'Superintelligence' that popularized AI existential risk, and key philosophical contributions...Quality: 25/100 - Author of "Superintelligence," early influence on Coefficient AI focus
- Eliezer YudkowskyPersonEliezer YudkowskyComprehensive biographical profile of Eliezer Yudkowsky covering his foundational contributions to AI safety (CEV, early problem formulation, agent foundations) and notably pessimistic views (>90% ...Quality: 35/100 - MIRI founder, recipient of early Coefficient AI safety grants