California SB 1047

Summary

SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, was California state legislation that would have required safety testing and liability measures for developers of the most powerful AI models.

The bill passed the California legislature but was vetoed by Governor Gavin NewsomPersonGavin NewsomGavin Newsom is the Governor of California. He vetoed SB 1047 on September 29, 2024, citing concerns about size-based rather than risk-based regulation, while simultaneously signing 18 other AI bil... on September 29, 2024.

SB 1047 was the most significant AI safety legislation attempted in the United States to date. Its passage through the legislature demonstrated growing political willingness to regulate frontier AI, while its veto illustrated the political challenges such regulation faces.

Quick Assessment

Key Links

What the Bill Proposed

Scope: "Covered Models"

The bill would have applied to AI models meeting any of these criteria:

Training Compute:

• Trained using >10^26 FLOP (floating-point operations)
• Approximately GPT-4.5/Claude 3 Opus scale or larger
• Threshold aligned with US Executive Order 14110↗🔗 webFenwick: Interesting Developments for Regulatory Thresholds of AI ComputeA law firm analysis relevant to AI governance practitioners tracking how compute-based regulatory thresholds are being developed; useful for understanding the legal landscape around frontier AI model oversight.A legal analysis from Fenwick examining evolving regulatory frameworks that use computational thresholds (e.g., FLOP counts) to define which AI systems trigger oversight require...governancepolicycomputeregulation+4Source ↗ on AI safety

Training Cost:

• Cost >$100 million to train
• Adjusted annually for inflation
• At current cloud compute prices, 10^26 FLOP costs approximately $70-100 million↗🔗 webCalifornia's SB 1047 Would Impose New Safety Requirements for Developers of Large-Scale AI ModelsLegal analysis from Morgan Lewis on SB 1047, a landmark but ultimately vetoed California bill representing one of the first major attempts at state-level frontier AI safety regulation; useful for understanding policy debates around AI governance.This Morgan Lewis legal analysis examines California's SB 1047, a proposed bill that would impose safety obligations on developers of large frontier AI models. The piece outline...governancepolicyregulationfrontier-models+5Source ↗ (Anthropic estimate)

Fine-tuned Models:

• Fine-tuning cost >$10 million
• Based on a covered model

Why these thresholds?

• Target only frontier models from well-resourced labs
• Exclude open-source models and academic research
• Align with international compute governance efforts (US EO, EU AI ActPolicyEU AI ActComprehensive overview of the EU AI Act's risk-based regulatory framework, particularly its two-tier approach to foundation models that distinguishes between standard and systemic risk AI systems. ...Quality: 55/100)

Provisions Comparison Table

Bill Structure and Enforcement Framework

The following diagram illustrates how SB 1047 would have regulated frontier AI development, from initial determination through deployment and enforcement:

flowchart TD
  A[AI Developer Plans Model Training] --> B{Will model meet<br/>covered thresholds?}
  B -->|Yes:<br/>>10^26 FLOP OR<br/>>$100M cost| C[Covered Model Path]
  B -->|No| Z[Exempt - No Requirements]

  C --> D[Pre-Training Requirements]
  D --> D1[Implement Shutdown Capability]
  D --> D2[Document Safety Protocol]
  D --> D3[Establish Cybersecurity]

  D1 & D2 & D3 --> E[Begin Training]

  E --> F[Pre-Deployment Testing]
  F --> F1[CBRN Capability Testing]
  F --> F2[Cyber Attack Testing<br/>>$500M damage threshold]
  F --> F3[Autonomous Operation Testing]
  F --> F4[Self-Improvement Testing]

  F1 & F2 & F3 & F4 --> G{Hazardous<br/>Capabilities<br/>Found?}

  G -->|Yes| H[Enhanced Safety Measures<br/>+ AG Notification]
  G -->|No| I[Standard Deployment]

  H --> J[Deployment with Monitoring]
  I --> J

  J --> K[Ongoing Obligations]
  K --> K1[Annual Third-Party Audit<br/>starting Jan 1, 2026]
  K --> K2[72-Hour Incident Reporting]
  K --> K3[Whistleblower Protections]
  K --> K4[5-Year Record Retention]

  K1 & K2 & K3 & K4 --> L{Compliance?}

  L -->|Yes| M[Affirmative Defense<br/>from Liability]
  L -->|No| N[Enforcement Actions]

  N --> N1[Attorney General Investigation]
  N1 --> N2{Violation<br/>Confirmed?}
  N2 -->|Yes| N3[Civil Penalties<br/>up to 10% training costs]
  N2 -->|No| M

  J --> O{Safety Incident<br/>Occurs?}
  O -->|Yes| P[72-Hour Report to AG]
  P --> N1
  O -->|No| L

  style C fill:#ffe6e6
  style D fill:#fff0e6
  style F fill:#fff8e6
  style G fill:#ffffe6
  style H fill:#ffe6e6
  style N fill:#ffcccc
  style N3 fill:#ff9999
  style M fill:#e6ffe6
  style Z fill:#e6f3ff

Key Enforcement Mechanisms:

The diagram shows three primary enforcement pathways in SB 1047:

1. Preventive Compliance Path (top): Developers who implement all requirements and pass audits receive affirmative defense from liability
2. Incident Response Path (bottom): Safety incidents trigger mandatory 72-hour reporting and Attorney General investigation
3. Penalty Path (right): Non-compliance or violation results in civil penalties up to 10% of training costs plus potential injunctive relief

The bill created overlapping accountability through third-party auditing (annual), whistleblower protections (continuous), and incident reporting (reactive), ensuring multiple mechanisms to detect non-compliance.

Core Requirements

1. Safety Testing Before Training

Pre-Training Requirements:

Developers must:

• Determine whether model will be a "covered model"
• Implement safety protocols before beginning training
• Establish shutdown procedures

Covered Model Determination:

If expected to meet thresholds:

• Document safety plan
• Prepare for testing requirements
• Establish compliance measures

2. Hazardous Capability Testing

Required Testing:

Before deployment or making available to third parties, test for:

Critical Harm Capabilities:

• Creation of chemical, biological, radiological, or nuclear weapons (CBRN)
• Mass casualty cyber attacks (>$500M damage or mass casualties)
• Autonomous operation and self-exfiltration
• Self-improvement and recursive self-modification

Testing Methods:

• Red-team testing
• Adversarial probing
• Capability evaluations
• Third-party auditing

Threshold: Model enables non-expert to cause mass casualties or >$500M in damage.

3. Safety and Security Protocol

Required Measures:

Developers must implement:

Cybersecurity:

• Protection of model weights from theft
• Secure infrastructure
• Incident response plans

Shutdown Capability:

• Full model shutdown ability
• Separate from safety fine-tuning
• Effective on all deployed instances

Ongoing Monitoring:

• Detection of hazardous use
• Capability creep tracking
• Post-deployment evaluation

Documentation:

• Written safety protocol
• Regular updates
• Public summary (redacted for security)

4. Whistleblower Protections

Employee Rights:

Protected disclosures about:

• Safety violations
• Unreasonable risk to public
• Non-compliance with the act

Prohibitions:

• Cannot retaliate against whistleblowers
• Cannot require non-disclosure preventing safety reports
• Civil penalties for violations

5. Frontier Model Division

New State Agency:

Created within California Government Operations Agency:

• Oversee compliance
• Receive safety protocols
• Investigate violations
• Issue guidance

Powers:

• Subpoena authority
• Civil penalty assessment
• Emergency orders

6. Liability Framework

Affirmative Defense:

Developers protected from liability if:

• Complied with all safety requirements
• Conducted reasonable testing
• Implemented safety protocols
• Acted in good faith

Strict Liability Removed:

Does NOT create automatic liability for harms; must prove negligence or non-compliance.

Attorney General Enforcement:

California AG can sue for:

• Violations of safety requirements
• Civil penalties up to 10% of training costs
• Injunctive relief

7. Compute Cluster Reporting

Reporting Requirement:

Owners of computing clusters with:

• 10^26 integer or floating-point operations per second

• Located in California

Must report to:

• Frontier Model Division
• Information about cluster
• Customers using cluster

Purpose: Track who has capability to train covered models.

Exemptions and Safe Harbors

Explicitly Exempted:

• Open-source models (unless developer makes $50M+/year from derivatives)
• Academic research
• Models below thresholds
• Government use

Safe Harbor:

• Compliance with safety requirements provides affirmative defense
• Good faith efforts protected

Limitations and Criticisms

Technical Limitations

Policy Limitations

Political Limitations

Criticisms from AI Safety Community

Some AI safety researchers opposed SB 1047 despite sharing its goals:

• Premature codification: Locking in specific requirements (10^26 FLOP, specific capability tests) before evaluation science matures could create compliance theater without actual safety
• Wrong target: Focus on model capabilities rather than deployment context may miss where risks actually materialize
• Opportunity cost: Political capital spent on failed state bill could have supported federal efforts or voluntary frameworks with broader buy-in
• Open-source concerns: Despite exemptions, uncertainty about derivative liability could chill beneficial open research

Path Through Legislature

Initial Introduction (February 2024)

Original Sponsors:

• Senator Scott WienerPersonScott WienerScott Wiener is a California State Senator representing District 11 (San Francisco). He is the author of SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, whi...↗🏛️ governmentSenator Scott WienerSenator Wiener's office is the origin of some of the most significant U.S. state-level AI safety legislation; relevant for tracking California AI regulatory developments that may influence national and global policy.Official webpage of California State Senator Scott Wiener, who represents San Francisco and is a prominent legislator on AI safety and technology policy. Wiener has authored sig...governancepolicyregulationstate-policy+4Source ↗ (D-San Francisco), representing District 11 (San Francisco tech corridor)
• Co-sponsored by AI safety organizations including Center for AI Safety
• Support from AI safety advocates and researchers
• Immediately opposed by major AI companies and some researchers, sparking fierce national debate
• Official bill text↗🏛️ governmentSB-1047 Safe and Secure Innovation for Frontier Artificial Intelligence Models ActSB 1047 was vetoed by California Governor Gavin Newsom in September 2024 despite passing the legislature; it remains highly influential as a template and reference point for ongoing AI governance efforts at state and federal levels.SB 1047 is California's 2024 landmark legislation requiring frontier AI model developers to implement safety protocols, maintain shutdown capabilities, and produce detailed safe...governancepolicyai-safetydeployment+5Source ↗ introduced February 7, 2024
• CSET Georgetown's early analysis noted the bill quickly became "a lightning rod for safety advocates and developers alike"

Amendment Process

Major Changes:

• Narrowed scope to truly frontier models (>10^26 FLOP or >$100M)
• Added safe harbors and affirmative defenses for compliant developers
• Reduced liability provisions (removed strict liability; kept negligence standard)
• Clarified open-source exemptions (unless developer earns >$50M/year from derivatives)
• Specified hazardous capabilities more precisely (CBRN, >$500M cyber damage)
• Removed Frontier Model Division↗🏛️ governmentSenator Wiener's Groundbreaking AI Bill Advances to Assembly Floor with AmendmentsThis official press release documents the amended version of California's SB 1047 as it advanced toward a final Assembly vote in August 2024, capturing the legislative compromises made in response to industry feedback on a high-profile AI safety regulation effort.California's SB 1047, a landmark AI safety bill requiring safety standards for large-scale AI developers, advanced to the Assembly floor in August 2024 with significant amendmen...governancepolicyregulationfrontier-models+4Source ↗ and criminal penalties in August 2024 amendments

Purpose of Amendments:

• Address industry concerns about overbreadth and compliance costs
• Balance innovation incentives with safety requirements
• Build bipartisan coalition for passage
• Respond to >50 stakeholder comments during committee process

Legislative Passage

August 29, 2024: Passed California Legislature (official vote records; CalMatters coverage)

• Assembly: 48-16 (75% approval)
• Senate: 32-1 (97% approval), concurrence 30-9
• Bipartisan support across party lines
• Most significant AI legislation to pass any US state legislature
• Represented months of amendments responding to >50 industry comments
• Final version removed criminal penalties and Frontier Model Division creation

Veto (September 29, 2024)

The veto was extensively covered by major outlets including the New York Times, Washington Post, The Verge, SF Standard, and TechCrunch, with policy analysis from CSET Georgetown↗🔗 web★★★★☆CSET GeorgetownMeta, OpenAI, and House Speaker Nancy Pelosi opposed the billCovers a pivotal 2024 US AI policy moment: the defeat of California's most ambitious AI safety legislation, relevant to understanding lobbying dynamics and the political landscape around AI governance.California Governor Gavin Newsom vetoed SB 1047 in September 2024, a landmark AI safety bill that would have imposed extensive safety protocols on large AI systems. Newsom cited...governancepolicyai-safetydeployment+2Source ↗ and Carnegie Endowment↗🔗 web★★★★☆Carnegie EndowmentA Heated California Debate Offers Lessons for AI Safety GovernanceRelevant to understanding how AI safety regulation is contested at the state level in the US; the SB 1047 veto is a key case study in the political economy of AI governance circa 2024.This Carnegie Endowment commentary analyzes California's SB 1047, a bipartisan AI safety bill that passed the legislature but was vetoed by Governor Newsom in September 2024. It...governancepolicyai-safetydeployment+3Source ↗. Zvi MowshowitzPersonZvi MowshowitzAI safety commentator and blogger. Author of 'Don't Worry About the Vase'. Former professional Magic: The Gathering player. Writes weekly AI updates.'s real-time analysis of the veto was widely shared in the AI safety community.

Governor Newsom's Rationale:

From Newsom's official veto message↗🏛️ governmentSB 1047 Veto MessageThis official veto message is a key primary source documenting the political and regulatory debate around compute-threshold-based AI safety legislation in California, reflecting real-world tensions between innovation, safety, and governance design choices.Governor Newsom vetoed California's SB 1047, which would have imposed safety requirements on large AI model developers based on computational thresholds. He argued the bill's si...governancepolicyai-safetyregulation+5Source ↗:

"While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions — so long as a large system deploys it."

Additional concerns: "Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047."

Specific Concerns:

• Focus on model size rather than deployment context
• Could stifle innovation in California's tech sector
• Regulatory approach not nuanced enough (described as not "informed by an empirical trajectory analysis")
• Preferred federal regulation given interstate nature of AI

Accompanying Actions:

Newsom simultaneously:

• Signed 18 other AI bills on narrower topics (deepfakes, discrimination, transparency)
• Called for federal AI legislation to address interstate nature of technology
• Committed to working with legislature on alternative approaches
• Convened expert panel including Fei-Fei LiPersonFei-Fei LiFei-Fei Li is the Sequoia Professor of Computer Science at Stanford University and co-director of the Stanford Human-Centered AI Institute (HAI). She created ImageNet, the large-scale visual recogn... (Stanford), Tino Cuéllar (Carnegie Endowment), and Jennifer Tour Chayes (UC Berkeley) to develop "empirical, science-based trajectory analysis"

Support and Opposition

Supporters

AI Safety Organizations:

• Center for AI SafetyOrganizationCenter for AI SafetyCAIS is a nonprofit research organization founded by Dan Hendrycks that has distributed compute grants to researchers, published technical AI safety papers including the representation engineering ...Quality: 42/100
• Future of Life InstituteOrganizationFuture of Life InstituteComprehensive profile of FLI documenting $25M+ in grants distributed (2015: $7M to 37 projects, 2021: $25M program), major public campaigns (Asilomar Principles with 5,700+ signatories, 2023 Pause ...Quality: 46/100
• AI safety researchers

Arguments:

• Frontier models pose catastrophic risks
• Industry self-regulation insufficient
• California can lead on AI safety
• Requirements are reasonable and achievable

Notable Individual Supporters:

• Yoshua BengioPersonYoshua BengioComprehensive biographical overview of Yoshua Bengio's transition from deep learning pioneer (Turing Award 2018) to AI safety advocate, documenting his 2020 pivot at Mila toward safety research, co...Quality: 39/100 (Turing Award winner, 2018)
• Geoffrey HintonPersonGeoffrey HintonComprehensive biographical profile of Geoffrey Hinton documenting his 2023 shift from AI pioneer to safety advocate, estimating 10-20% extinction risk in 5-20 years. Covers his media strategy, poli...Quality: 42/100 (Turing Award winner 2018, "Godfather of AI")
• Stuart RussellPersonStuart RussellStuart Russell (born 1962) is a British computer scientist and UC Berkeley professor who co-authored the dominant AI textbook 'Artificial Intelligence: A Modern Approach' (used in over 1,500 univer...Quality: 30/100 (UC Berkeley professor, author of leading AI textbook)
• Max TegmarkPersonMax TegmarkComprehensive biographical profile of Max Tegmark covering his transition from cosmology to AI safety advocacy, his role founding the Future of Life Institute, and his controversial Mathematical Un...Quality: 63/100 (MIT professor, founder of Future of Life Institute)
• Elon MuskPersonElon MuskComprehensive profile of Elon Musk's role in AI, documenting his early safety warnings (2014-2017), OpenAI founding and contentious departure, xAI launch and funding history, Neuralink BCI developm...Quality: 38/100 (xAIOrganizationxAIComprehensive profile of xAI covering its founding by Elon Musk in 2023, rapid growth to $230B valuation and $3.8B revenue, development of Grok models, and controversial 'truth-seeking' safety appr...Quality: 48/100 CEO, publicly endorsed the bill)
• 113+ current and former employees of OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100, Google DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100, AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials ($380B valuation, $14B ARR at Series G growing to $19B by March 2026), safety research (Constitutional AI, mechanistic interpretability...Quality: 74/100, MetaOrganizationMeta AI (FAIR)Comprehensive organizational profile of Meta AI covering $66-72B infrastructure investment (2025), LLaMA model family (1B+ downloads), and transition from FAIR research lab to product-focused GenAI...Quality: 51/100, and xAIOrganizationxAIComprehensive profile of xAI covering its founding by Elon Musk in 2023, rapid growth to $230B valuation and $3.8B revenue, development of Grok models, and controversial 'truth-seeking' safety appr...Quality: 48/100 (September 9, 2024 letter↗📖 reference★★★☆☆WikipediaSafe and Secure Innovation for Frontier Artificial Intelligence Models Act (Wikipedia)SB 1047 was a major flashpoint in AI policy debates in 2024; understanding its provisions and veto provides essential context for ongoing discussions about how to regulate frontier AI development at state and federal levels.Wikipedia article covering California's SB 1047, a landmark proposed state-level AI safety bill that would have imposed safety obligations on developers of large frontier AI mod...governancepolicyregulationfrontier-models+4Source ↗ to Governor Newsom)

Opponents

Major AI Companies:

• OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100 (initially opposed; later neutral) — Senator Wiener publicly responded to OpenAI's opposition, calling it inconsistent (covered by TechCrunch)
• AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials ($380B valuation, $14B ARR at Series G growing to $19B by March 2026), safety research (Constitutional AI, mechanistic interpretability...Quality: 74/100 (opposed initially; called final version "benefits likely outweigh costs" but remained uncertain; later published The Case for Targeted Regulation outlining their preferred regulatory framework)
• Google/DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100 (opposed)
• MetaOrganizationMeta AI (FAIR)Comprehensive organizational profile of Meta AI covering $66-72B infrastructure investment (2025), LLaMA model family (1B+ downloads), and transition from FAIR research lab to product-focused GenAI...Quality: 51/100 (strongly opposed)
• Combined market value of opposing companies: >$3 trillion

Arguments:

• Stifles innovation in California's $200+ billion AI industry
• Drives development out of California (threatening 300,000+ tech jobs)
• Premature to regulate models that don't yet exist
• Better to focus on use cases than model capabilities (size-based vs. risk-based regulation)
• Federal regulation more appropriate for interstate technology

Venture Capital:

• Y CombinatorOrganizationY CombinatorY Combinator is the world's most prominent startup accelerator, founded in 2005. It has funded over 4,000 startups including Airbnb, Stripe, Dropbox, and many AI companies. YC publicly opposed SB 1...
• Andreessen HorowitzOrganizationAndreessen Horowitz (a16z)Andreessen Horowitz (a16z) is a major venture capital firm founded in 2009, known for large investments in AI companies. The firm actively opposed SB 1047, publishing detailed analysis arguing the ...
• Others concerned about startup ecosystem impact

Some Researchers:

• Yann LeCunPersonYann LeCunComprehensive biographical profile of Yann LeCun documenting his technical contributions (CNNs, JEPA), his ~0% AI extinction risk estimate, and his opposition to AI safety regulation including SB 1...Quality: 41/100 (MetaOrganizationMeta AI (FAIR)Comprehensive organizational profile of Meta AI covering $66-72B infrastructure investment (2025), LLaMA model family (1B+ downloads), and transition from FAIR research lab to product-focused GenAI...Quality: 51/100, Turing Award winner)
• Andrew NgPersonAndrew NgAndrew Ng is one of the most prominent figures in AI education and research. He co-founded Google Brain, served as Chief Scientist at Baidu, and co-founded Coursera. His online courses have taught ... (Stanford, Google Brain co-founder)
• Fei-Fei LiPersonFei-Fei LiFei-Fei Li is the Sequoia Professor of Computer Science at Stanford University and co-director of the Stanford Human-Centered AI Institute (HAI). She created ImageNet, the large-scale visual recogn... (Stanford)

Concerns:

• Open-source implications despite exemptions
• Compliance costs for startups
• Regulatory overreach
• Vague standards

Labor and Progressive Groups:

• Some supported
• Some concerned it didn't address labor impacts enough

Why It Was Vetoed

Stated Reasons (Governor Newsom)

Size-Based vs. Risk-Based:

• Bill focuses on model size (compute/cost) not deployment risks
• Small models in high-risk contexts not covered
• Large models in benign contexts over-regulated

Innovation Concerns:

• California is hub of AI development
• Regulation could drive companies elsewhere
• Startups face compliance burdens

Federal Action Preferable:

• AI transcends state borders
• National framework more appropriate
• International coordination needed

Political Analysis

Industry Pressure:

• Major AI companies lobbied heavily against
• Economic arguments about California's AI ecosystem
• Threat of relocation

Presidential Politics:

• Biden administration developing AI policy
• Harris (VP, former CA Senator) in presidential race
• National Democratic messaging on tech

Tactical Considerations:

• Newsom signed 18 other AI bills simultaneously
• Positioned as pro-innovation, pro-safety balance
• Left door open for future iteration

Lack of Coalition:

• Many Democrats skeptical — Nancy Pelosi and CA Congressional Democrats urged Newsom to veto (see Pelosi's official statement)
• Republicans opposed
• Labor not fully engaged
• Insufficient grassroots pressure

Unstated Factors (Analysis)

Economic:

• California tech industry contributes $200+ billion annually to state GDP
• AI-focused companies employ 300,000+ workers in California
• Competing jurisdictions (Texas, Florida, international) actively recruiting AI companies
• Tech industry contributes 15-20% of California's general fund revenue
• Estimated compliance costs for SB 1047: $10-50M per covered model annually (industry estimates)

Policy:

• Precedent-setting implications
• Uncertainty about effectiveness
• Implementation challenges

Political:

• Presidential election dynamics
• Tech industry relationships
• Future political ambitions

Implications for AI Safety Regulation

What SB 1047 Demonstrated

Political Will Exists:

• Bipartisan legislative passage showed AI safety resonates
• Not just fringe concern but mainstream political issue
• Legislators willing to regulate despite industry opposition

Industry Opposition is Formidable:

• Even safety-focused companies (AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials ($380B valuation, $14B ARR at Series G growing to $19B by March 2026), safety research (Constitutional AI, mechanistic interpretability...Quality: 74/100) opposed
• Economic arguments effective
• Innovation framing powerful

Federal vs. State Tension:

• AI is inherently interstate and international
• State-level regulation faces jurisdictional limits
• But federal action is slow

Details Matter:

• Size-based vs. risk-based framing was central
• Specific thresholds and requirements heavily debated
• Implementation details crucial to political viability

Lessons for Future Efforts

What Worked

Focused Scope:

• Targeting only frontier models built support
• Exemptions for open-source and research
• Concrete thresholds (compute, cost)

Safety Framing:

• Catastrophic risk resonated
• Whistleblower protections popular
• Bipartisan appeal

Expert Endorsement:

• Turing Award winners lending credibility
• Technical community engagement

What Didn't Work

Industry Consensus:

• Even safety-concerned labs opposed
• Economic arguments effective
• Innovation framing won

Implementation Clarity:

• Vague enforcement mechanisms
• Uncertainty about compliance costs
• Questions about Frontier Model Division capacity

Coalition Building:

• Labor not fully engaged
• Grassroots support limited
• Competing priorities on left

Future Regulatory Approaches

State Level

Narrower Bills:

• Focus on specific harms (deepfakes, discrimination)
• Deployment context rather than model capabilities
• Procurement standards

Coordination:

• Multi-state coordination
• Uniform standards
• Regional compacts

California Iteration:

• Newsom committed to continued dialogue
• Future versions possible
• Refined approach incorporating feedback

Federal Level

Legislation:

• Comprehensive AI safety bill
• Build on Executive Order
• Bipartisan framework

Challenges:

• Congressional gridlock
• Lobbying pressure
• Competing priorities

International

Coordination Imperative:

• AI development global
• Race to the bottom risk
• Need for international standards

Precedents:

• EU AI Act as model
• UK approach
• Multilateral frameworks

Impact on AI Safety Movement

EA and AI Safety Organization Involvement

SB 1047 represented the most direct engagement of the EA-aligned AI safety community with state-level legislation. Zvi Mowshowitz's comprehensive guide to SB 1047 became the single most widely-read analysis in the AI safety community, while Scott AlexanderPersonScott AlexanderScott Alexander is a psychiatrist and writer behind Astral Codex Ten (ACT), one of the most widely-read blogs in the rationalist and effective altruism communities. His writing on AI risk, predicti...'s post-veto account of the campaign provided an inside perspective on the advocacy effort:

Direct organizational support:

• Center for AI Safety (CAIS)OrganizationCenter for AI SafetyCAIS is a nonprofit research organization founded by Dan Hendrycks that has distributed compute grants to researchers, published technical AI safety papers including the representation engineering ...Quality: 42/100 — Dan HendrycksPersonDan HendrycksComprehensive reference biography of Dan Hendrycks (CAIS director), covering his academic career (GELU, MMLU, OOD detection), CAIS founding and funding (including $6.5M FTX, Open Philanthropy/Coeff...Quality: 19/100 was a key organizer of support, leveraging CAIS's network of researchers and the 2023 Statement on AI Risk signatories
• Future of Life InstituteOrganizationFuture of Life InstituteComprehensive profile of FLI documenting $25M+ in grants distributed (2015: $7M to 37 projects, 2021: $25M program), major public campaigns (Asilomar Principles with 5,700+ signatories, 2023 Pause ...Quality: 46/100 — provided analysis and advocacy through their policy team
• The bill's core framing (catastrophic risk from frontier models, compute thresholds) closely aligned with EA-adjacent thinking on AI governance

The 113-employee letter: One of the most striking developments was the September 2024 letter to Governor Newsom↗📖 reference★★★☆☆WikipediaSafe and Secure Innovation for Frontier Artificial Intelligence Models Act (Wikipedia)SB 1047 was a major flashpoint in AI policy debates in 2024; understanding its provisions and veto provides essential context for ongoing discussions about how to regulate frontier AI development at state and federal levels.Wikipedia article covering California's SB 1047, a landmark proposed state-level AI safety bill that would have imposed safety obligations on developers of large frontier AI mod...governancepolicyregulationfrontier-models+4Source ↗ signed by 113+ current and former employees of OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100, Google DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100, AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials ($380B valuation, $14B ARR at Series G growing to $19B by March 2026), safety research (Constitutional AI, mechanistic interpretability...Quality: 74/100, MetaOrganizationMeta AI (FAIR)Comprehensive organizational profile of Meta AI covering $66-72B infrastructure investment (2025), LLaMA model family (1B+ downloads), and transition from FAIR research lab to product-focused GenAI...Quality: 51/100, and xAIOrganizationxAIComprehensive profile of xAI covering its founding by Elon Musk in 2023, rapid growth to $230B valuation and $3.8B revenue, development of Grok models, and controversial 'truth-seeking' safety appr...Quality: 48/100. This was significant because:

• It showed safety-minded employees at frontier labs were willing to publicly break with their employers' positions
• The letter specifically cited catastrophic risk concerns aligned with EA safety arguments
• It demonstrated grassroots support within the AI industry that went beyond the organized safety community

EA Forum and LessWrong debate: SB 1047 generated extensive discussion within EA online communities, with coverage from Vox's Future Perfect and other EA-adjacent outlets. The 80,000 Hours podcast episode with Nathan Calvin provided a deep dive into the bill's development and the safety community's role, while the CAIS AI Safety Newsletter covered the bill's passage and the veto aftermath. Zvi Mowshowitz also published an early analysis on LessWrong that shaped initial community opinion. The debate centered on:

• Whether the bill's specific mechanisms (compute thresholds, capability testing) were well-designed
• Whether state-level legislation was the right venue for frontier AI regulation
• The strategic implications of Anthropic's mixed position — a safety-focused lab declining to fully support a safety bill
• Whether the political capital spent on SB 1047 could have been better deployed elsewhere

Positive Effects

Mainstream Attention:

• SB 1047 brought frontier AI risk into public discourse beyond the EA/AI safety community
• Generated extensive media coverage from NYT, Washington Post, Wired, Politico, and others
• The campaign was documented in a 31-minute documentary film and analyzed in multi-perspective podcast debates
• Created a concrete policy proposal that translated abstract x-risk concerns into legislative language

Overton Window Shift:

• Made AI regulation a mainstream political issue rather than a fringe concern
• Future regulatory proposals are now measured against SB 1047 as a reference point
• The bill's passage through the legislature (Assembly 48-16, Senate 32-1) proved that AI safety legislation can achieve overwhelming bipartisan support

Community Building:

• Built coalitions between AI safety researchers, policy advocates, and sympathetic legislators
• Developed political skills and institutional knowledge within the AI safety community
• The SB 1047 campaign infrastructure directly contributed to SB 53's eventual passage

Negative Effects

Backlash:

• Some researchers became more skeptical of legislative approaches to AI safety
• "Regulatory capture" accusations from both industry (regulation benefits incumbents) and open-source advocates (harms small developers)
• Polarization between "regulate now" and "wait for better science" camps within AI safety

Movement Division:

• The most consequential split was Anthropic's position — a company founded explicitly for AI safety opposing (then hedging on) a safety bill
• Tensions between deontological ("we must act now") and consequentialist ("will this actually reduce risk?") approaches to AI governance
• Open-source AI community alienated despite the bill's exemptions

Political Capital:

• The veto was framed by opponents as evidence that AI safety regulation is politically unviable
• Industry lobby appeared emboldened, with subsequent state AI bills facing increased opposition
• However, the SB 53 outcome complicates this narrative — the veto was a setback, not a defeat

Strategic Lessons

For the AI safety movement:

• Industry alignment has limits: Even safety-focused labs will oppose regulation that threatens their competitive position or operational flexibility
• State-level regulation works as an Overton window tool: SB 1047's passage made federal proposals seem more moderate by comparison
• Iterative beats comprehensive: The SB 1047 → SB 53 arc suggests incremental approaches achieve more durable outcomes than ambitious first attempts
• Employee voice matters: The 113-employee letter demonstrated that individual AI researchers can influence policy debates independent of their employers

Strategic Debates

Should SB 1047 Have Been Pursued?

Arguments For:

• Only way to test political viability
• Built coalition and momentum
• Shifted discourse even in defeat

Arguments Against:

• Premature; should have built more support first
• Better to focus on federal action
• Antagonized potential allies

What Should Come Next?

Double Down:

• Refine and reintroduce
• Build broader coalition
• Address veto concerns

Pivot to Federal:

• Focus energy on Congress
• Support Executive Order implementation
• International coordination

Focus on Narrower Wins:

• Procurement standards
• Use-case specific regulation
• Voluntary frameworks

Build Power:

• Grassroots organizing
• Labor coalition
• Public education

Lobbying and Industry Tactics

The opposition to SB 1047 involved sophisticated political tactics from the technology industry:

Lobbyist connections to Newsom:

• Andreessen HorowitzOrganizationAndreessen Horowitz (a16z)Andreessen Horowitz (a16z) is a major venture capital firm founded in 2009, known for large investments in AI companies. The firm actively opposed SB 1047, publishing detailed analysis arguing the ... hired Axiom Advisors, led by Jason Kinney — a long-time Newsom advisor and dining companion (the "French Laundry connection"). a16z paid $39,750 for April-June 2024 lobbying work. a16z also backed the StopSB1047.com opposition campaign site.
• Y CombinatorOrganizationY CombinatorY Combinator is the world's most prominent startup accelerator, founded in 2005. It has funded over 4,000 startups including Airbnb, Stripe, Dropbox, and many AI companies. YC publicly opposed SB 1... engaged in formal California lobbying for the first time, hiring Platinum Advisors led by Darius Anderson, another firm with close Newsom ties.

Push polling: The California Chamber of Commerce commissioned a push poll (August 9-12, 2024) using loaded language about "small startup companies" paying "tens of millions in fines from state bureaucrats." Results: 28% support, 46% oppose, 26% neutral. The poll was conducted by Adam Rosenblatt of Bold Decision. Despite the framing, the bill's actual exemptions for small companies and open-source developers were not mentioned in the poll questions.

Campaign contribution context: Google was described as the biggest contributor to certain California representatives, with nearly $1 million in total campaign contributions, though these were not SB 1047-specific.

State-Level Copycat Legislation

SB 1047's influence extended beyond California even after its veto, inspiring frontier AI legislation in other states. As MIT Technology Review reported, New York legislators explicitly sought to "pick up the pieces" of the vetoed California bill:

The New York RAISE Act became the first state law directly comparable to SB 1047's ambitions, though California's SB 53 took a lighter-touch transparency approach. This divergence illustrates the ongoing policy debate: whether frontier AI regulation should focus on mandatory safety requirements (SB 1047 / NY RAISE) or transparency and reporting (SB 53).

Successor Legislation: SB 53 and the Expert Panel

Newsom's Expert Panel

Following the SB 1047 veto, Governor Newsom convened the Joint California Policy Working Group on AI Frontier Models, comprising leading AI researchers and policy experts:

• Fei-Fei LiPersonFei-Fei LiFei-Fei Li is the Sequoia Professor of Computer Science at Stanford University and co-director of the Stanford Human-Centered AI Institute (HAI). She created ImageNet, the large-scale visual recogn... (Stanford) — co-chair
• Tino Cuéllar (Carnegie Endowment for International Peace) — co-chair
• Jennifer Tour Chayes (UC Berkeley, Dean of Computing)

The panel released its recommendations in June 2025, which:

• Rejected a compute-threshold-only approach (the core SB 1047 mechanism)
• Endorsed a transparency-first framework with mandatory incident reporting
• Recommended creation of CalCompute, a state computing resource for AI safety research
• Advocated for risk-based rather than size-based regulatory triggers

SB 53: The Successor Bill

SB 53PolicyCalifornia SB 53California SB 53 represents the first U.S. state law specifically targeting frontier AI safety through transparency requirements, incident reporting, and whistleblower protections, though it makes ...Quality: 73/100, also authored by Senator Scott WienerPersonScott WienerScott Wiener is a California State Senator representing District 11 (San Francisco). He is the author of SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, whi..., was signed into law by Governor NewsomPersonGavin NewsomGavin Newsom is the Governor of California. He vetoed SB 1047 on September 29, 2024, citing concerns about size-based rather than risk-based regulation, while simultaneously signing 18 other AI bil... on September 29, 2025 — exactly one year after the SB 1047 veto.

The 18 Companion Bills

When vetoing SB 1047, Newsom simultaneously signed 18 other AI bills, signaling support for narrower regulation:

Privacy, Data, and Transparency:

Protection Against Harmful AI Content:

Elections:

Public Sector and Healthcare:

SB 1047 to SB 53 Arc as Policy Case Study

The SB 1047 → expert panel → SB 53 trajectory represents an important case study in AI governance:

1. Ambitious bill demonstrates political feasibility — SB 1047 passed the legislature with overwhelming majorities, proving AI safety regulation is not politically toxic
2. Veto channels energy into expert process — rather than killing the effort, the veto redirected it through a technical review panel
3. Scaled-back version achieves enactment — SB 53 secured the transparency and reporting infrastructure that future enforcement-focused legislation can build on
4. Iterative approach validated — the one-year arc from veto to signature suggests that incremental state-level regulation may be more durable than comprehensive approaches

Technical and Policy Debates

Size-Based vs. Risk-Based Regulation

Size-Based (SB 1047 Approach):

Pros:

• Objective, measurable thresholds
• Targets most capable models
• Easier to enforce
• Aligns with international compute governance

Cons:

• Doesn't capture deployment context
• Could miss dangerous applications of smaller models
• Algorithmic efficiency makes thresholds obsolete

Risk-Based (Newsom's Preference):

Pros:

• Focuses on actual harm potential
• Context-appropriate
• Adapts to changing technology

Cons:

• Harder to define and measure
• Enforcement challenges
• Potentially broader scope (privacy, fairness, etc.)
• Risk assessment subjective

Synthesis Possible:

• Combination of both approaches
• Size thresholds trigger risk assessments
• Deployment context determines requirements

Liability Questions

SB 1047 Approach:

• Affirmative defense for compliance
• Attorney General enforcement
• Civil penalties

Debate:

• Too much liability deters innovation?
• Too little fails to ensure safety?
• Who should bear costs of AI harms?

Alternative Approaches:

• Strict liability with caps
• Insurance requirements
• Tiered liability based on precautions
• No-fault compensation schemes

Open Source Implications

SB 1047 Exemption:

• Open-source models exempt unless developer profits >$50M from derivatives

Concerns Raised:

• Could still chill open-source development
• Uncertainty about liability
• Derivative work tracking difficult

Counter-Arguments:

• Exemption was broad
• Open-source not inherently safe
• Need some oversight of powerful models

Ongoing Debate:

• How to encourage open research while managing risks
• Different models for different risk levels
• Role of open-source in AI safety ecosystem

Comparison to Other Policies

vs. US Executive Order

The compute thresholds in SB 1047 were deliberately aligned↗🔗 webFenwick: Interesting Developments for Regulatory Thresholds of AI ComputeA law firm analysis relevant to AI governance practitioners tracking how compute-based regulatory thresholds are being developed; useful for understanding the legal landscape around frontier AI model oversight.A legal analysis from Fenwick examining evolving regulatory frameworks that use computational thresholds (e.g., FLOP counts) to define which AI systems trigger oversight require...governancepolicycomputeregulation+4Source ↗ with Biden's Executive Order 14110.

Similarities:

• Compute thresholds (10^26 FLOP for training)
• Safety testing requirements for CBRN risks
• Focus on frontier models only
• Developer reporting obligations

Differences:

• SB 1047 had enforcement teeth (civil penalties up to 10% of training costs, AG lawsuits)
• EO has broader scope (government use, competition policy, immigration for AI talent)
• SB 1047 state-level mandatory law; EO federal executive action (can be rescinded)
• SB 1047 required shutdown capability (unique provision)
• SB 1047 included third-party auditing requirement (EO relies on voluntary compliance)

Relationship:

• SB 1047 would have complemented EO with state-level enforcement
• State enforcement of federal principles with local adaptation
• Potential model for other states considering AI legislation
• Analysis from legal firms↗🔗 webGibson Dunn: Regulating the Future -- Eight Key Takeaways from SB 1047Legal analysis from Gibson Dunn on California's SB 1047, a landmark but ultimately vetoed AI safety bill; useful reference for understanding state-level AI governance debates and the regulatory challenges of overseeing frontier AI models.Richard Manfredi (2024)Gibson Dunn law firm analyzes California's SB 1047, a significant AI safety bill that would have imposed safety requirements on developers of large frontier AI models, which was...governancepolicyregulationfrontier-models+4Source ↗ noted SB 1047 went further than EO on liability

vs. EU AI Act

EU Act:

• Risk categories for deployed systems
• Broader scope (not just frontier models)
• Binding regulation with large fines

SB 1047:

• Narrower focus on frontier models
• More specific technical requirements (shutdown, testing)
• State-level vs. EU-wide

Lessons:

• EU's comprehensiveness politically difficult in US
• SB 1047's focused approach still failed
• Suggests US regulation will be patchwork

vs. Voluntary Commitments

Industry Commitments:

• No enforcement
• Self-defined standards
• Flexible and adaptive

SB 1047:

• Mandatory requirements
• State enforcement
• Specific standards

Debate:

• Is voluntary compliance sufficient?
• Does regulation stifle beneficial innovation?
• Can industry self-regulate emerging risks?

Career and Research Implications

Policy Careers

Lessons Learned:

• Understanding legislative process crucial
• Coalition building essential
• Technical expertise must translate to policy

Opportunities:

• State-level AI policy growing
• Need for policy entrepreneurs
• Legislative staff positions

Research Questions

Regulatory Design:

• How to balance innovation and safety?
• What thresholds are appropriate?
• How to make regulation adaptive?

Political Economy:

• Industry influence on regulation
• Public opinion on AI risk
• Coalition formation strategies

Technical:

• Measuring model capabilities
• Shutdown mechanisms
• Audit methodologies

Movement Building

Strategic Questions:

• When to push for regulation vs. build support?
• How to engage industry productively?
• Building public constituency

Skills Needed:

• Political strategy
• Coalition management
• Communications
• Policy design

Sources

Primary Documents

• California Legislature: SB-1047 Bill Text (Enrolled Version) - Full enrolled bill text
• California Legislature: SB-1047 Overview↗🏛️ governmentSB-1047 Safe and Secure Innovation for Frontier Artificial Intelligence Models ActSB 1047 was vetoed by California Governor Gavin Newsom in September 2024 despite passing the legislature; it remains highly influential as a template and reference point for ongoing AI governance efforts at state and federal levels.SB 1047 is California's 2024 landmark legislation requiring frontier AI model developers to implement safety protocols, maintain shutdown capabilities, and produce detailed safe...governancepolicyai-safetydeployment+5Source ↗ - Official bill navigation and legislative history
• California Legislature: SB-1047 Bill Status - Legislative status and timeline
• California Legislature: SB-1047 Bill Analysis - Committee analyses from both chambers
• California Legislature: SB-1047 Votes - Official vote records (Assembly 48-16, Senate 32-1)
• Governor Newsom's Veto Message (PDF)↗🏛️ governmentSB 1047 Veto MessageThis official veto message is a key primary source documenting the political and regulatory debate around compute-threshold-based AI safety legislation in California, reflecting real-world tensions between innovation, safety, and governance design choices.Governor Newsom vetoed California's SB 1047, which would have imposed safety requirements on large AI model developers based on computational thresholds. He argued the bill's si...governancepolicyai-safetyregulation+5Source ↗ - Official veto statement, September 29, 2024
• The California Report on Frontier AI Policy (June 2025) - Expert panel final report
• California Assembly Privacy and Consumer Protection Committee Analysis↗🏛️ governmentCalifornia Assembly Privacy and Consumer Protection Committee AnalysisOfficial California legislative committee analysis of SB 1047 (2024), a landmark state-level AI safety bill that sparked major debate about whether subnational governments should regulate frontier AI; ultimately vetoed by Governor Newsom but influential in shaping AI governance discourse.This California Assembly Privacy and Consumer Protection Committee analysis examines SB 1047, which would impose comprehensive regulatory requirements on developers of frontier ...governancepolicyregulationfrontier-models+6Source ↗ - Detailed bill analysis, June 18, 2024
• Pelosi: Statement in Opposition to California Senate Bill 1047 - Official congressional opposition statement

News Coverage and Analysis

• New York Times: California Governor Vetoes AI Safety Bill - Coverage of the veto decision
• Washington Post: California AI Bill SB 1047 Vetoed by Newsom - Analysis of the veto and its implications
• CalMatters: Newsom vetoes major California artificial intelligence bill↗🔗 webCalMatters: Newsom vetoes major California artificial intelligence billThis news article covers a pivotal moment in US AI governance: Newsom's veto of SB 1047 in September 2024, which was the most prominent attempt at state-level frontier AI safety regulation and sparked broad debate about the appropriate locus and design of AI oversight.California Governor Gavin Newsom vetoed SB 1047, a landmark AI safety bill that would have imposed safety requirements on developers of large frontier AI models. The veto was a ...governancepolicyregulationfrontier-models+3Source ↗ - Comprehensive coverage of veto decision
• NPR: California Gov. Newsom vetoes AI safety bill that divided Silicon Valley↗🔗 webNPR: California Gov. Newsom vetoes AI safety bill that divided Silicon ValleyKey news event in AI governance: Newsom's veto of SB 1047 in September 2024 was a pivotal moment for US state-level AI safety regulation, reflecting ongoing tensions between safety advocates and the tech industry.California Governor Gavin Newsom vetoed SB 1047, a landmark AI safety bill that would have imposed safety requirements on developers of large frontier AI models. The veto highli...governancepolicyai-safetydeployment+4Source ↗ - Context on industry division
• TechCrunch: California's legislature just passed AI bill SB 1047↗🔗 web★★★☆☆TechCrunchTechCrunch: California's legislature just passed AI bill SB 1047SB 1047 was a major 2024 state-level AI safety bill in California; understanding the debate around it is important context for AI governance discussions, even though Governor Newsom ultimately vetoed the bill in September 2024.TechCrunch covers the California legislature's passage of SB 1047, a landmark AI safety bill targeting large frontier models. The bill imposes safety obligations on developers o...governancepolicyregulationfrontier-models+4Source ↗ - Coverage of legislative passage
• The Verge: California Governor Gavin Newsom Vetoes AI Safety Bill SB 1047 - Tech press coverage of veto
• Wired: Over 100 AI Employees Wrote to Gov. Newsom About SB 1047 - Coverage of the 113-employee letter
• Politico: Pelosi Opposes California AI Regulation - Congressional Democratic opposition
• Vox Future Perfect: California AI Safety Bill SB 1047 - Analysis from effective altruism-adjacent outlet
• Carnegie Endowment: All Eyes on Sacramento: SB 1047 and the AI Safety Debate↗🔗 web★★★★☆Carnegie EndowmentCarnegie Endowment: All Eyes on Sacramento: SB 1047 and the AI Safety DebateA Carnegie Endowment policy analysis of California's SB 1047, a landmark attempted state-level frontier AI safety bill that was vetoed in 2024; useful context for understanding U.S. sub-federal AI governance debates and the politics of frontier model regulation.This Carnegie Endowment analysis examines California's SB 1047 bill, which sought to impose safety requirements on developers of large frontier AI models. It explores the policy...governancepolicyai-safetyfrontier-models+4Source ↗ - Policy analysis
• Fortune: California AI Bill SB 1047 Fierce Debate - Feature on the national debate over the bill (July 2024)
• SF Standard: Gavin Newsom Vetoes Controversial AI Safety Bill - Local coverage of the veto decision
• TechCrunch: Gov. Newsom Vetoes California's Controversial AI Bill SB 1047 - Tech press veto coverage
• TechCrunch: OpenAI's Opposition to California's AI Law Makes No Sense, Says State Senator - Coverage of Wiener's response to OpenAI
• CalMatters: AI Safety Bill Passes California Legislature - Coverage of legislative passage (August 2024)
• MIT Technology Review: A New York Legislator Wants to Pick Up the Pieces of the Dead California AI Bill - Coverage of SB 1047's influence on other states (January 2025)

Legal and Technical Analysis

• Morgan Lewis: California's SB 1047 Would Impose New Safety Requirements↗🔗 webCalifornia's SB 1047 Would Impose New Safety Requirements for Developers of Large-Scale AI ModelsLegal analysis from Morgan Lewis on SB 1047, a landmark but ultimately vetoed California bill representing one of the first major attempts at state-level frontier AI safety regulation; useful for understanding policy debates around AI governance.This Morgan Lewis legal analysis examines California's SB 1047, a proposed bill that would impose safety obligations on developers of large frontier AI models. The piece outline...governancepolicyregulationfrontier-models+5Source ↗ - Legal analysis of requirements
• Gibson Dunn: Regulating the Future: Eight Key Takeaways from California's SB 1047↗🔗 webGibson Dunn: Regulating the Future -- Eight Key Takeaways from SB 1047Legal analysis from Gibson Dunn on California's SB 1047, a landmark but ultimately vetoed AI safety bill; useful reference for understanding state-level AI governance debates and the regulatory challenges of overseeing frontier AI models.Richard Manfredi (2024)Gibson Dunn law firm analyzes California's SB 1047, a significant AI safety bill that would have imposed safety requirements on developers of large frontier AI models, which was...governancepolicyregulationfrontier-models+4Source ↗ - Post-veto analysis
• Orrick: California Looks to Regulate Cutting-Edge Frontier AI Models: 5 Things to Know↗🔗 webOrrick: California Looks to Regulate Cutting-Edge Frontier AI Models: 5 Things to KnowSB 1047 was a high-profile 2024 California bill that sparked major debate in the AI industry; it was ultimately vetoed by Governor Newsom in September 2024, but remains influential as a model for future AI legislation.Orrick law firm provides a legal analysis of California's SB 1047, a landmark bill proposing safety requirements for developers of large frontier AI models. The piece outlines k...governanceregulationpolicyfrontier-models+4Source ↗ - Technical requirements breakdown
• DLA Piper: California's SB-1047: Understanding the Safe and Secure Innovation for Frontier AI Act↗🔗 webDLA Piper: California's SB-1047: Understanding the Safe and Secure Innovation for Frontier AI ActLegal analysis from a major law firm on California's SB-1047, relevant for organizations developing frontier AI systems navigating emerging US state-level AI safety regulation; the bill was ultimately vetoed by Governor Newsom in September 2024.DLA Piper provides a legal analysis of California's SB-1047, which mandates safety determinations before training frontier AI models, establishes compliance requirements, and cr...governancepolicyregulationfrontier-models+4Source ↗ - Early analysis
• Fenwick: Technological Challenges for Regulatory Thresholds of AI Compute↗🔗 webFenwick: Interesting Developments for Regulatory Thresholds of AI ComputeA law firm analysis relevant to AI governance practitioners tracking how compute-based regulatory thresholds are being developed; useful for understanding the legal landscape around frontier AI model oversight.A legal analysis from Fenwick examining evolving regulatory frameworks that use computational thresholds (e.g., FLOP counts) to define which AI systems trigger oversight require...governancepolicycomputeregulation+4Source ↗ - Analysis of compute thresholds
• Stanford HAI: Analysis of the Safe and Secure Innovation for Frontier AI Models Act - Academic policy analysis
• Lawfare: SB 1047 and the Future of Frontier AI Safety Regulation - Legal and policy analysis
• CSET Georgetown: Governor Newsom Vetoes Sweeping AI Regulation SB 1047↗🔗 web★★★★☆CSET GeorgetownMeta, OpenAI, and House Speaker Nancy Pelosi opposed the billCovers a pivotal 2024 US AI policy moment: the defeat of California's most ambitious AI safety legislation, relevant to understanding lobbying dynamics and the political landscape around AI governance.California Governor Gavin Newsom vetoed SB 1047 in September 2024, a landmark AI safety bill that would have imposed extensive safety protocols on large AI systems. Newsom cited...governancepolicyai-safetydeployment+2Source ↗ - Veto analysis from Georgetown's Center for Security and Emerging Technology
• Carnegie Endowment: California SB 1047 AI Safety Bill Veto Lessons↗🔗 web★★★★☆Carnegie EndowmentA Heated California Debate Offers Lessons for AI Safety GovernanceRelevant to understanding how AI safety regulation is contested at the state level in the US; the SB 1047 veto is a key case study in the political economy of AI governance circa 2024.This Carnegie Endowment commentary analyzes California's SB 1047, a bipartisan AI safety bill that passed the legislature but was vetoed by Governor Newsom in September 2024. It...governancepolicyai-safetydeployment+3Source ↗ - Post-veto lessons and policy analysis
• Lawfare: California's Proposed SB 1047 Would Be a Major Step Forward for AI Safety↗📄 paper★★★☆☆arXiv[2311.04850] Rethinking Benchmark and Contamination for Language Models with Rephrased SamplesThis paper addresses data contamination in large language models by demonstrating that simple rephrasing techniques can bypass existing decontamination methods, highlighting critical vulnerabilities in benchmark trustworthiness and evaluation integrity.Shuo Yang, Wei-Lin Chiang, Lianmin Zheng et al. (2023)183 citationsThis paper reveals critical vulnerabilities in current data decontamination practices for large language models. The authors demonstrate that standard string-matching decontamin...capabilitiestrainingevaluationllmSource ↗ - Balanced pros/cons analysis of the bill's provisions
• CSET Georgetown: California AI Bill Becomes a Lightning Rod - Early analysis from Georgetown's Center for Security and Emerging Technology

Senator Wiener's Office

• Senator Wiener: Groundbreaking AI Bill Advances to Assembly Floor↗🏛️ governmentSenator Wiener's Groundbreaking AI Bill Advances to Assembly Floor with AmendmentsThis official press release documents the amended version of California's SB 1047 as it advanced toward a final Assembly vote in August 2024, capturing the legislative compromises made in response to industry feedback on a high-profile AI safety regulation effort.California's SB 1047, a landmark AI safety bill requiring safety standards for large-scale AI developers, advanced to the Assembly floor in August 2024 with significant amendmen...governancepolicyregulationfrontier-models+4Source ↗ - Official statement on amendments
• Senator Wiener: Bipartisan Vote, Senate Passes Landmark AI Safety Bill↗🏛️ governmentSenator Wiener: Bipartisan Vote, Senate Passes Landmark AI Safety BillSB 1047 was a landmark but controversial California AI safety bill in 2024; it was ultimately vetoed by Governor Newsom, but the legislative debate significantly shaped discussions around frontier AI regulation at the state level.The California Senate passed SB 1047, Senator Scott Wiener's landmark AI safety bill, with bipartisan support. The bill establishes safety requirements for developers of large f...ai-safetygovernancepolicydeployment+5Source ↗ - Official statement on passage
• Lawfare Daily Podcast: State Senator Scott WienerPersonScott WienerScott Wiener is a California State Senator representing District 11 (San Francisco). He is the author of SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, whi... on SB 1047↗🔗 web★★★★☆LawfareLawfare Daily Podcast: State Senator Scott Wiener on SB 1047This 2024 podcast provides direct insight from the sponsor of California's SB 1047, a landmark and contested state-level AI safety bill targeting frontier model developers, useful for understanding the political dynamics of AI governance efforts.California State Senator Scott Wiener joins Lawfare to discuss SB 1047, his 'Safe and Secure Innovation for Frontier Artificial Intelligence Models' bill. The conversation cover...governancepolicyregulationfrontier-models+4Source ↗ - In-depth interview
• Senator Wiener Responds to OpenAI Opposition to SB 1047 - Official response to OpenAI's public opposition

Industry Perspectives

• Andreessen Horowitz: What You Need to Know About SB 1047↗🔗 webAndreessen Horowitz: What You Need to Know About SB 1047This is a prominent VC firm's industry-perspective critique of SB 1047, useful for understanding opposition arguments to state-level AI safety legislation; represents a16z's policy stance rather than neutral analysis.Andreessen Horowitz presents its analysis and opposition to California's SB 1047 AI safety bill, arguing that the legislation would stifle innovation, harm open-source AI develo...governancepolicyregulationstate-policy+4Source ↗ - Venture capital perspective
• Safe and Secure AI: Letter to YC & a16z↗🔗 webSafe and Secure AI: Letter to YC & a16zThis letter is part of the public debate around AI safety legislation (likely California SB 1047), representing the pro-regulation side responding to prominent VC firms' lobbying against such bills.An open letter responding to Y Combinator and Andreessen Horowitz's opposition to AI safety legislation, arguing that safety regulations for frontier AI models are necessary and...ai-safetygovernancepolicyfrontier-models+3Source ↗ - Response from supporters
• Brookings: Misrepresentations of California's AI safety bill↗🔗 web★★★★☆Brookings InstitutionBrookings: Misrepresentations of California's AI safety billPublished during the contentious 2024 debate over California SB 1047, this Brookings piece is relevant for understanding arguments in favor of state-level AI safety regulation and the rhetorical landscape surrounding frontier model governance.A Brookings Institution analysis defending California's AI safety bill (SB 1047) against what the authors characterize as misleading criticisms from opponents. The piece examine...ai-safetygovernancepolicyregulation+3Source ↗ - Defense of bill against criticism
• Anthropic: The Case for Targeted Regulation - Post-veto regulatory framework advocating for targeted rather than broad AI regulation
• StopSB1047.com - a16z-backed opposition campaign site against the bill

EA Community and AI Safety Analysis

• Zvi Mowshowitz: Guide to SB 1047 - Comprehensive analysis of the bill's provisions and implications (August 2024)
• Zvi Mowshowitz: Newsom Vetoes SB 1047 - Analysis of the veto decision and its aftermath (October 2024)
• Scott Alexander: SB 1047: Our Side of the Story - Inside account of the advocacy campaign from an AI safety perspective (October 2024)
• 80,000 Hours Podcast: Nathan Calvin on SB 1047 - Deep dive into the bill's development and the safety community's role
• CAIS AI Safety Newsletter #40: California AI Legislation - Coverage of SB 1047's passage through the legislature
• CAIS AI Safety Newsletter #42: Newsom Vetoes SB 1047 - Analysis of the veto from an AI safety perspective
• Zvi Mowshowitz: On the Proposed California SB 1047 (LessWrong) - Early community analysis that shaped initial opinion
• The Inside View: SB 1047 Documentary - 31-minute documentary film on the bill's journey
• Cognitive Revolution: Final Analysis on CA's AI Bill SB 1047 - Multi-perspective podcast debate with Nathan Calvin, Dean Ball, and Steve Newman
• EA Forum: SB 1047 Analysis - Effective altruism community discussion and analysis

Reference

• Wikipedia: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act↗📖 reference★★★☆☆WikipediaSafe and Secure Innovation for Frontier Artificial Intelligence Models Act (Wikipedia)SB 1047 was a major flashpoint in AI policy debates in 2024; understanding its provisions and veto provides essential context for ongoing discussions about how to regulate frontier AI development at state and federal levels.Wikipedia article covering California's SB 1047, a landmark proposed state-level AI safety bill that would have imposed safety obligations on developers of large frontier AI mod...governancepolicyregulationfrontier-models+4Source ↗ - Overview and timeline
• FastDemocracy: SB 1047 Bill Tracking - Legislative tracking and timeline
• SafeSecureAI.org - Senator Wiener's campaign site for SB 1047