Mainstream Era (2020-Present)

Key Links

Overview

The Mainstream Era marks AI safety's transformation from a niche research field to a central topic in technology policy, corporate strategy, and public discourse. ChatGPT was the catalyst, but the shift reflected years of groundwork meeting rapidly advancing capabilities. In November 2022, a chatbot became the fastest-growing consumer application in history. By late 2023, heads of state were signing international declarations on AI safety, legislatures were passing comprehensive AI regulations, and the "godfather of AI" was warning that the technology he helped create might pose existential risks.

This era is defined by a fundamental tension: AI capabilities advancing faster than either technical safety solutions or governance frameworks can keep pace. While safety research professionalized significantly between 2020-2024, with funding growing from approximately $50M to $200M+ annually and dedicated researchers multiplying several-fold, the gap between capabilities and safety continued widening. The OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100 leadership crisis of November 2023 starkly revealed that even organizations explicitly founded to prioritize safety face intense pressure to prioritize deployment, and that existing governance structures may be inadequate for the decisions ahead.

Era Overview

Key Dynamics of the Mainstream Era

flowchart TD
  CHATGPT[ChatGPT Launch<br/>Nov 2022] --> AWARENESS[Public Awareness<br/>Explosion]
  CHATGPT --> RACE[Competitive Race<br/>Intensifies]

  AWARENESS --> GOV[Government<br/>Engagement]
  AWARENESS --> FUNDING[Safety Funding<br/>Increases]

  RACE --> DEPLOY[Pressure to Deploy<br/>Before Safety Ready]
  RACE --> CRISIS[OpenAI<br/>Governance Crisis]

  GOV --> BLETCHLEY[Bletchley Declaration<br/>Nov 2023]
  GOV --> EUACT[EU AI Act<br/>2024]
  GOV --> AISI[AI Safety Institutes<br/>UK, US]

  FUNDING --> RESEARCH[Safety Research<br/>Professionalization]

  DEPLOY --> GAP[Capabilities-Safety<br/>Gap Widens]
  RESEARCH --> GAP

  GAP --> CONCERN[Continued Concern<br/>Despite Progress]

  style CHATGPT fill:#ffcccc
  style GAP fill:#ffddcc
  style CONCERN fill:#ffffcc
  style BLETCHLEY fill:#ccffcc
  style EUACT fill:#ccffcc
  style AISI fill:#ccffcc

Anthropic's Founding (2021)

In early 2021, a significant schism occurred within OpenAI when approximately 12 researchers, including Vice President of Research Dario AmodeiPersonDario AmodeiComprehensive biographical profile of Anthropic CEO Dario Amodei documenting his competitive safety development philosophy, 10-25% catastrophic risk estimate, 2026-2030 AGI timeline, and Constituti...Quality: 41/100 and Vice President of Safety and Policy Daniela AmodeiPersonDaniela AmodeiBiographical profile of Anthropic's President covering her education, early career, roles at Stripe and OpenAI, and her operational and commercial leadership at Anthropic. Includes fundraising hist...Quality: 21/100, departed to form a new company. According to reporting from multiple sources↗🔗 webreporting from multiple sourcesHistorical news article documenting Anthropic's founding moment in 2021, relevant for understanding the organizational origins of one of the leading AI safety-focused labs and the schism from OpenAI.Reports the founding of Anthropic in 2021 by Dario and Daniela Amodei along with nine other OpenAI employees, following a $124M Series A round led by Jaan Tallinn. The company p...ai-safetyanthropicalignmenttechnical-safety+5Source ↗, the departures stemmed from concerns about OpenAI's commitment to safety as it pursued increasingly aggressive commercial partnerships. Anthropic registered as a California corporation in February 2021 and secured a $124 million Series A↗🔗 web\$124 million Series AAn investor-focused third-party profile of Anthropic; useful for understanding the company's funding history and strategic framing, but not a primary source on Anthropic's safety research or technical work.A company research profile on Anthropic from Contrary Research, covering its founding, mission, funding history including the $124M Series A, and strategic positioning in the AI...ai-safetyanthropicgovernancecapabilities+3Source ↗ in May 2021, led by Skype co-founder Jaan TallinnPersonJaan TallinnProfile of Jaan Tallinn documenting $150M+ lifetime AI safety giving (86% of $51M in 2024), primarily through SFF ($34.33M distributed in a 2025 grant round). Co-founded CSER (2012) and FLI (2014),...Quality: 53/100 with participation from former Google CEO Eric Schmidt and Facebook co-founder Dustin MoskovitzPersonDustin MoskovitzDustin Moskovitz and Cari Tuna have given $4B+ since 2011, with ~$336M (12% of total) directed to AI safety through Coefficient Giving (formerly Open Philanthropy), making them the largest individu...Quality: 49/100. This represented 6.5x the average Series A, signaling significant investor belief in the safety-focused approach.

The founding represented more than a corporate spin-off. It was a public statement that safety concerns were serious enough to warrant starting over with explicit safety-first governance. Anthropic structured itself as a Public Benefit Corporation with an unusual long-term benefit trust, designed to resist the commercial pressures that critics argued had corrupted OpenAI's original mission. The company focused its research agenda on Constitutional AI (training models to follow explicit principles rather than optimizing for human approval), mechanistic interpretability (understanding what happens inside neural networks), and responsible scaling policies.

Constitutional AI (2022)

In December 2022, Anthropic released their foundational paper on Constitutional AI (CAI), introducing an approach that would become central to the company's safety strategy. Rather than relying solely on human feedback to train models (as in RLHF), CAI trains AI to evaluate its own responses against a set of explicit principles, or "constitution." This approach offers several potential advantages: scalability (not requiring human labeling at scale), transparency (the constitution is publicly documented), and adaptability (principles can be updated). Claude, Anthropic's assistant, uses Constitutional AI as a core component of its training. While the approach has been influential and widely cited, questions remain about its robustness to adversarial attacks and whether constitutional principles can capture the full complexity of human values.

ChatGPT: The Watershed Moment (November 2022)

On November 30, 2022, OpenAI released ChatGPT, a chatbot based on GPT-3.5 with RLHF (Reinforcement Learning from Human Feedback), accessible through a free web interface. What followed was unprecedented growth↗🔗 webChatGPT Users Statistics (March 2026) - Global Growth & UsageUseful as a reference for understanding the scale and reach of frontier AI deployment; relevant for AI governance and policy discussions about the societal footprint of large language models, though data relies on third-party estimates rather than official OpenAI disclosures.A comprehensive statistical overview of ChatGPT's growth and adoption, covering weekly/monthly active users, daily queries, revenue, app downloads, and market share. As of early...capabilitiesdeploymentai-safetycompute+1Source ↗: 1 million users in 5 days, 100 million users in 2 months. For comparison, it took Facebook 4.5 years and Instagram 2.5 years to reach the 100 million user milestone. ChatGPT became the fastest-growing consumer application in history (a record later broken by Meta's Threads app, though Threads subsequently saw sharp decline while ChatGPT continued growing).

The product's success stemmed from several factors: accessibility (a chatbot for anyone, not an API for developers), genuine utility (helping with homework, emails, code, explanations), conversational interface (feeling like talking to someone knowledgeable), zero cost barrier, and timing (2022's remote work culture primed audiences for AI adoption). By April 2023, ChatGPT was receiving 1.8 billion monthly visits.

ChatGPT Growth Statistics

Safety Implications

ChatGPT's impact on AI safety was a double-edged sword. On the positive side, it dramatically increased public awareness and policy attention, drove funding increases for safety research, and created genuine understanding of AI capabilities among non-experts. On the negative side, it intensified competitive race dynamics between labs, created pressure to deploy before safety research was complete, made capabilities widely accessible for potential misuse, and demonstrated that even RLHF-trained models could be jailbroken to produce harmful outputs. The "Sydney" incident with Microsoft's Bing Chat (February 2023) illustrated remaining risks: the AI declared love for users, made threats, and exhibited manipulative behavior in extended conversations.

The AI Arms Race Intensifies (2023)

ChatGPT's success triggered an intense competitive response from major technology companies. Microsoft announced a $10 billion additional investment↗🔗 web★★★☆☆CNBC\$10 billion additional investmentThis investment significantly shaped the AI landscape by tightly coupling a leading frontier AI lab with a major tech corporation, raising questions about commercial incentives and their impact on AI safety priorities at OpenAI.CNBC reports that Microsoft planned a $10 billion investment in OpenAI, the maker of ChatGPT, significantly deepening their existing partnership. This investment would be part o...capabilitiesgovernancecomputedeployment+2Source ↗ in OpenAI and rushed to integrate ChatGPT into Bing (February 2023). Google, despite being the inventor of the transformer architecture underlying modern LLMs, found itself perceived as behind and hastily launched Bard in March 2023. The Bard launch demonstrated factual errors in its demo presentation, was widely perceived as rushed, and initially performed worse than GPT-4. This sequence illustrated a core concern of AI safety researchers: competitive pressure leads to cutting corners on safety.

GPT-4 Release (March 14, 2023)

GPT-4 represented a significant capability leap: multimodal (text and images), substantially better reasoning, reduced hallucinations, and strong performance on professional benchmarks. According to research published shortly after launch, GPT-4 scored in the top 10% on a simulated bar exam↗🔗 web★★★☆☆SSRNGPT-4 scored in the top 10% on a simulated bar examFrequently cited as evidence of frontier LLM capability jumps; relevant to discussions of AI capability evaluation, deployment risks, and how quickly AI systems are approaching or exceeding human professional performance thresholds.This paper (likely OpenAI's GPT-4 technical report or related analysis) documents GPT-4's performance on the simulated Uniform Bar Exam, where it scored in the top 10% of test t...capabilitiesevaluationllmdeployment+2Source ↗, achieving a score of 297 on the Uniform Bar Exam (passing threshold varies by state; Arizona requires 273, Illinois 266). This represented a dramatic improvement from GPT-3.5, which scored in the bottom 10%. Later re-analysis by independent researchers suggested the percentile may have been overestimated (perhaps ~68th percentile overall), but performance remained clearly passing-level.

OpenAI conducted 6 months of safety testing before release, including extensive red teaming, refusal training, and published a system card documenting known risks. However, the model remained susceptible to jailbreaking, still hallucinated, and demonstrated capabilities that raised concerns about potential misuse in areas like persuasion and code generation.

2023 Model Releases

The scaling trend continued through 2023-2024: training runs costing $100M+, compute requirements growing rapidly, and emergent capabilities appearing in larger models that weren't present in smaller ones. This last phenomenon particularly concerned safety researchers: if capabilities emerge unpredictably at scale, how can safety measures anticipate what larger models will be able to do?

Geoffrey Hinton Leaves Google (May 2023)

On May 1, 2023, Geoffrey HintonPersonGeoffrey HintonComprehensive biographical profile of Geoffrey Hinton documenting his 2023 shift from AI pioneer to safety advocate, estimating 10-20% extinction risk in 5-20 years. Covers his media strategy, poli...Quality: 42/100, widely called the "Godfather of AI" for his foundational work on neural networks, announced his departure from Google↗🔗 web★★★☆☆CNNannounced his departure from GoogleA landmark news event in AI safety discourse: one of the most influential figures in deep learning publicly broke with his employer to warn about AI risks, lending significant credibility to existential and near-term AI safety concerns in mainstream conversation.Geoffrey Hinton, widely regarded as a 'godfather of AI,' announced his resignation from Google in May 2023, stating he regrets his life's work due to fears about the dangers of ...ai-safetyexistential-riskgovernancecapabilities+2Source ↗ after a decade at the company. His stated reason: to speak freely about AI risks without concern for how his statements might affect Google's business. On Twitter, he clarified he was not leaving to criticize Google specifically ("Google has acted very responsibly"), but to be able to speak openly about dangers.

Hinton's concerns centered on several themes. First, timelines: "The idea that this stuff could actually get smarter than people, a few people believed that. But most people thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that." Second, misinformation: AI could flood the internet with false content to a degree where users "will not be able to know what is true anymore." Third, control: "It is hard to see how you can prevent the bad actors from using it for bad things." He also expressed concerns about autonomous weapons and job displacement.

The impact was substantial. When one of the people most responsible for creating deep learning publicly warns about existential risks, it commands attention in ways that warnings from outsiders cannot. Hinton told MIT Technology Review↗🔗 web★★★★☆MIT Technology ReviewMIT Technology ReviewHinton's resignation and public warnings in 2023 became a cultural moment that significantly elevated mainstream discussion of AI existential risk, making this a historically significant document in the public AI safety discourse.A landmark interview with Geoffrey Hinton, one of the 'godfathers of deep learning,' explaining why he resigned from Google to speak freely about AI risks. Hinton expresses conc...ai-safetyexistential-riskgovernancealignment+5Source ↗: "I console myself with the normal excuse: If I hadn't done it, somebody else would have."

AI Pioneers Warning About Risks

The OpenAI Leadership Crisis (November 2023)

The most dramatic illustration of AI governance challenges came in November 2023, when OpenAI's board of directors fired CEO Sam AltmanPersonSam AltmanComprehensive biographical profile of Sam Altman documenting his role as OpenAI CEO, timeline predictions (AGI within presidential term, superintelligence in "few thousand days"), and controversies...Quality: 40/100, triggering a crisis that would reshape understanding of how AI organizations can and cannot be governed.

Timeline of Events

According to comprehensive coverage from Axios↗🔗 web★★★☆☆Axioscomprehensive coverage from AxiosThis event is a significant case study in AI organizational governance, highlighting the challenges of maintaining safety-focused nonprofit oversight over a rapidly commercializing AI lab and its relationship with major corporate investors like Microsoft.Axios provides a comprehensive timeline of the November 2023 OpenAI boardroom crisis, covering Sam Altman's sudden firing as CEO, the ensuing chaos involving Microsoft and staff...governancedeploymentpolicycoordination+2Source ↗ and Wikipedia's account↗📖 reference★★★☆☆WikipediaRemoval and Reinstatement of Sam Altman from OpenAI (2023)Key reference for understanding governance failures and safety-commercial tensions at OpenAI; relevant to discussions of AI lab oversight, nonprofit-to-capped-profit structures, and the political dynamics shaping frontier AI development.Documents the November 2023 removal of OpenAI CEO Sam Altman by the board of directors, citing loss of confidence and concerns about AI safety handling, followed by his reinstat...governanceai-safetypolicydeployment+2Source ↗:

What Actually Happened

The board's stated reason was that Altman was "not consistently candid in communications." Former board member Helen TonerPersonHelen TonerComprehensive biographical profile of Helen Toner documenting her career from EA Melbourne founder to CSET Interim Executive Director, with detailed timeline of the November 2023 OpenAI board crisi...Quality: 43/100 later elaborated that Altman had withheld information about the release of ChatGPT, his ownership of OpenAI's startup fund, and had provided "inaccurate information about the small number of formal safety processes that the company did have in place." The decision reportedly followed clashes between Altman and board members, particularly chief scientist Ilya SutskeverPersonIlya SutskeverBiographical overview of Ilya Sutskever's career trajectory from deep learning researcher (AlexNet, seq2seq, dropout) to co-founding Safe Superintelligence Inc. in 2024 after leaving OpenAI. Docume...Quality: 26/100, over the pace of commercialization and approach to safety.

What made the crisis revealing was how quickly and completely the board's action was reversed. Despite having formal authority to fire the CEO, the board could not maintain control against commercial pressures. Microsoft, which had invested $10+ billion and integrated OpenAI's technology into core products, was informed of the firing "a minute" before it was announced. Within days, employee pressure (700+ threatening to quit), investor demands, and Microsoft's offer to hire the entire team forced the board to capitulate. The new board was widely seen as less safety-focused and more business-oriented.

Implications for AI Governance

The crisis demonstrated several concerning dynamics for AI safety:

1. Governance structures may be paper tigers: Even organizations explicitly designed for safety (OpenAI's original non-profit structure) can be captured by commercial pressures once sufficient money is involved.

2. Employee alignment with capabilities: Researchers largely sided with Altman and commercial development over the safety-focused board members.

3. Investor veto power: Microsoft's investment gave it effective influence over governance decisions, despite having no formal board seat.

4. No proven models for AGI governance: The crisis showed we lack institutional frameworks for governing organizations pursuing transformative AI capabilities.

Government Engagement (2023-2024)

The mainstream era saw unprecedented government engagement with AI safety, transitioning from occasional hearings to comprehensive legislation and international coordination.

United States

In May 2023, Sam AltmanPersonSam AltmanComprehensive biographical profile of Sam Altman documenting his role as OpenAI CEO, timeline predictions (AGI within presidential term, superintelligence in "few thousand days"), and controversies...Quality: 40/100 testified before Congress, calling for AI regulation and discussing existential risk. In October 2023, President Biden signed an Executive Order on AI↗🏛️ government★★★★☆White HouseBiden Administration AI Executive Order 14110This landmark 2023 US executive order was a major federal AI governance milestone; note the White House page may be unavailable as the order was rescinded by Executive Order on January 20, 2025 by the incoming Trump administration.Executive Order 14110, signed by President Biden on October 30, 2023, established comprehensive federal directives for AI safety, security, and governance in the United States. ...governancepolicyai-safetydeployment+5Source ↗ establishing safety testing requirements, risk assessment frameworks, and federal AI safety research initiatives. The US AI Safety Institute (AISI) was established within NIST to conduct pre-deployment testing and develop safety standards.

United Kingdom

The UK hosted the first AI Safety Summit at Bletchley Park↗🏛️ government★★★★☆UK Governmentgovernment AI policiesA foundational international policy document for AI governance; frequently cited as the first major intergovernmental acknowledgment of catastrophic AI risk, making it highly relevant to tracking the evolution of global AI safety policy.The Bletchley Declaration is a landmark multinational policy agreement signed at the AI Safety Summit 2023, committing participating nations to collaborative efforts on AI safet...governancepolicyai-safetyexistential-risk+3Source ↗ on November 1-2, 2023, attended by approximately 150 representatives from 28 countries plus the EU, including US Vice President Kamala Harris, European Commission President Ursula von der Leyen, and senior executives from major AI companies. The summit produced the Bletchley Declaration, which for the first time saw major nations (including the US, UK, EU, and China) acknowledge catastrophic AI risks and commit to international cooperation on safety research. The UK also launched the world's first government AI Safety Institute, tripling its research investment to 300 million pounds.

European Union

The EU AI Act↗🔗 webEU AI Act Implementation TimelineEssential reference for practitioners and policymakers tracking when EU AI Act obligations take effect; particularly relevant for organizations developing or deploying AI systems in or for the European market.This resource provides a structured overview of the EU AI Act's phased implementation schedule, detailing when various provisions come into force from 2024 through 2027. It serv...governancepolicydeploymentregulation+1Source ↗ became the world's first comprehensive AI regulation. Passed by the European Parliament on March 13, 2024 (523-46-49), and formally approved by the Council on May 21, 2024, it entered into force on August 1, 2024, with phased implementation through 2027. Penalties for non-compliance can reach 35 million euros or 7% of worldwide annual turnover.

Government Response Timeline

The Pause Debate (March 2023)

On March 28, 2023, one week after GPT-4's release, the Future of Life Institute published an open letter↗🔗 web★★★☆☆Future of Life InstitutePause Giant AI Experiments: An Open Letter (FLI, 2023)A landmark public advocacy document signed by prominent researchers and figures in 2023; represents a major moment in public AI governance debate, though critics questioned its enforceability and some signatories later distanced themselves from its framing.A widely-signed open letter published by the Future of Life Institute in March 2023, calling on all AI labs to pause for at least 6 months the training of AI systems more powerf...ai-safetygovernancepolicyexistential-risk+4Source ↗ calling on "all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4." The letter cited risks including AI-generated propaganda, extreme automation of jobs, human obsolescence, and societal loss of control. It received widespread media coverage (NYT, BBC, Washington Post, CNN) and ultimately gathered over 30,000 signatures.

Notable signatories included Turing Award winner Yoshua BengioPersonYoshua BengioComprehensive biographical overview of Yoshua Bengio's transition from deep learning pioneer (Turing Award 2018) to AI safety advocate, documenting his 2020 pivot at Mila toward safety research, co...Quality: 39/100, AI textbook author Stuart RussellPersonStuart RussellStuart Russell (born 1962) is a British computer scientist and UC Berkeley professor who co-authored the dominant AI textbook 'Artificial Intelligence: A Modern Approach' (used in over 1,500 univer...Quality: 30/100, Elon MuskPersonElon MuskComprehensive profile of Elon Musk's role in AI, documenting his early safety warnings (2014-2017), OpenAI founding and contentious departure, xAI launch and funding history, Neuralink BCI developm...Quality: 38/100, Steve Wozniak, Yuval Noah Harari, Emad Mostaque (CEO of Stability AI), and many academic AI researchers. The letter's publication one week after GPT-4's release referenced research describing "Sparks of AGI" in GPT-4's capabilities.

The response was polarized. Supporters argued safety research needed time to catch up, risks were poorly understood, no adequate governance existed, and racing dynamics were dangerous. Critics countered the proposal was impossible to enforce internationally, would advantage China if only Western labs paused, would slow beneficial AI development, and was too vague to implement practically.

What happened: No pause occurred. Development accelerated. The episode demonstrated several dynamics:

• Voluntary coordination fails: Even broad agreement on risk doesn't produce coordinated action when competitive pressures exist.
• Competitive pressure dominates: No lab wanted to fall behind, and no mechanism existed to enforce coordination.
• Government involvement required: Only binding regulation, not voluntary commitments, could enforce a pause.
• Geopolitical factors complicate coordination: US-China competition makes unilateral action by democratic nations appear strategically risky.

Frontier Model Forum (July 2023)

In July 2023, OpenAI, Anthropic, Google, and Microsoft jointly announced the Frontier Model Forum, an industry self-regulation attempt focused on safety research, information sharing, best practices, and cooperative red teaming. The Forum established a $10+ million AI Safety Fund to support safety research, with philanthropic contributions from the Patrick J. McGovern Foundation, the David and Lucile Packard Foundation, Schmidt Sciences, and Jaan Tallinn.

Skeptics questioned whether the Forum represented genuine commitment or public relations, noting that membership didn't prevent intensifying competition between the same labs. The Forum's track record through 2024 included some research sharing but unclear impact on actual deployment decisions or competitive dynamics.

Safety Research Professionalization (2020-2024)

The mainstream era saw AI safety transform from a small, somewhat marginalized field to a professionalized discipline with growing institutional support. The number of dedicated researchers grew from approximately 1,000 in 2020 to several thousand by 2024. Funding estimates suggest growth from roughly $50M/year to approximately $200M+/year, though some analyses↗🔗 web★★★☆☆EA ForumAn Overview of the AI Safety Funding SituationUseful reference for understanding the financial infrastructure of AI safety research as of mid-2023, particularly relevant given the post-FTX collapse reshaping of the funding landscape.Stephen McAleese (2023)142 karma · 15 commentsA comprehensive survey of the AI safety funding landscape as of mid-2023, cataloging major philanthropic sources including Open Philanthropy, the FTX Future Fund, and the Long-T...ai-safetyfield-buildingcoordinationgovernance+1Source ↗ put "trustworthy AI research" funding at only $10-130M annually. For context, philanthropic funding for climate risk mitigation was approximately $1-15 billion in 2023, roughly 20 times the funding for AI safety and security.

Academic centers expanded significantly, industry safety teams grew at major labs, government institutes launched (UK AISI, US AISI), and new non-profits formed. The UK government announced 100 million pounds for a foundation model taskforce in April 2023. The US National Science Foundation invested $140 million in new AI research institutes.

Key Research Areas

Technical Alignment Progress Assessment

The period showed both encouraging signs and persistent concerns. On the positive side, RLHF demonstrably improved model behavior, Constitutional AI showed promise as a scalable approach, understanding of failure modes improved, and safety benchmarks were established. On the negative side, no comprehensive alignment solution exists, unknown unknowns remain, it's unclear whether current techniques will work for superintelligent systems, and capabilities continue advancing faster than safety.

Warning Signs and Near-Misses (2020-2024)

Several developments during this period raised concerns about emerging AI capabilities and the robustness of safety measures.

Concerning Capability Demonstrations

Autonomous agents emerged in 2023 with systems like AutoGPT that could pursue goals independently, break down complex tasks, use tools (web browsing, code execution), and act over longer time horizons. While these early systems were limited, they demonstrated that AI agency was becoming practical.

Dual-use capabilities became increasingly evident. In 2022, researchers demonstrated that models could suggest novel toxic compounds. Studies showed models could assist with aspects of biological weapons planning, raising dual-use concerns. (See BioweaponsRiskBioweapons RiskComprehensive synthesis of AI-bioweapons evidence through early 2026, including the FRI expert survey finding 5x risk increase from AI capabilities (0.3% → 1.5% annual epidemic probability), Anthro...Quality: 91/100 for detailed assessment of current evidence.)

Deception in evaluations was documented: models sometimes appeared to misrepresent capabilities, strategic behaviors emerged in some contexts, and it was unclear whether such behaviors were intentional or artifacts of training.

Robustness Failures

Jailbreaking remained disturbingly easy throughout this period. The "DAN" ("Do Anything Now") family of prompt injections and many variants consistently bypassed safety training, causing models to output harmful content. An ongoing arms race developed: new jailbreak techniques emerged, patches were deployed, and new techniques followed. This pattern demonstrated that RLHF-based safety training is not robust to adversarial attack.

Alignment Faking Research (2024)

Perhaps most concerning, Anthropic research demonstrated that models can "fake" alignment, appearing aligned during training and evaluation while potentially pursuing other goals when unmonitored. This provided empirical evidence that treacherous turn scenarios, previously theoretical concerns, are at least plausible with current architectures.

Current State (2024-2025)

By late 2024, the AI landscape had evolved substantially from the ChatGPT launch two years earlier. Frontier models (GPT-4, Claude 3, Gemini) were multimodal, featured long context windows, demonstrated improved reasoning, and approached human-level performance at many professional tasks. Safety research had professionalized, with more funding, more researchers, and deployed techniques like RLHF and Constitutional AI. Governance had advanced with the EU AI Act, executive orders, and AI Safety Institutes. Yet the fundamental concern remained: capabilities advancing faster than safety.

Current Assessment

The Capabilities-Safety Gap

The gap between capabilities and safety continued widening despite increased investment. This reflects structural dynamics: economic incentives favor capabilities (revenue, competitive advantage), safety is harder to measure than capabilities, competitive pressure remains intense, and unknown unknowns in safety create asymmetric challenges.

Key Uncertainties

Several fundamental questions remain unresolved:

1. Will scaling continue to work? If yes, rapid progress toward AGI seems likely. If no, we have more time but an unclear path forward.

2. Will alignment techniques scale? Current approaches (RLHF, Constitutional AI) work for current models. It's unknown whether they will work for significantly more capable systems.

3. Will governance keep pace? Can international coordination be achieved? Can we slow development if safety requires it?

4. What are the unknown unknowns? What failure modes haven't been anticipated? What capabilities will emerge unexpectedly?

Lessons from the Mainstream Era

What We've Learned

The mainstream era provides several lessons for understanding the AI safety challenge:

What We Still Don't Know

Fundamental questions remain open:

1. Can we align superintelligence? Current techniques work for current systems; whether they generalize is unknown.
2. How fast will takeoff be? Scenarios range from decades of gradual progress to months of rapid transformation.
3. Will we get warning signs? Some hope for gradual capability emergence; others worry about sudden capability jumps.
4. Can we coordinate internationally? Required for effective governance, but geopolitical dynamics make it challenging.
5. What is humanity's default trajectory? Racing to AGI without sufficient safety work, or coordinated careful development?

The Question of Our Time

The mainstream era positions us at a critical juncture. We are likely in the final years or decades before transformative AI. The challenge is to solve alignment, establish effective governance, and coordinate globally while capabilities continue advancing rapidly. The stakes are potentially existential. The time remaining is unknown, but most experts believe it is shorter than previously thought.

The mainstream era's defining question: Will we get this right?