State-Space Models / Mamba

Key Links

Overview

State-Space Models (SSMs), particularly the Mamba architecture developed by Albert Gu (CMU) and Tri Dao (Princeton), represent a fundamentally different approach to sequence modeling than transformers. Instead of the pairwise attention mechanism (quadratic O(n^2) complexity), SSMs use structured state-space dynamics derived from continuous-time systems theory, achieving linear O(n) complexity in sequence length.

The efficiency gains are substantial: Mamba achieves 5x higher inference throughput than comparably-sized transformers and the Mamba-3B model matches Transformer-6B perplexity while being 40% cheaper to run. On the Long Range Arena benchmark, the foundational S4 model achieved 80.48% average accuracy—the first architecture to solve the Path-X task requiring reasoning over 16,384 tokens—compared to less than 60% for all transformer baselines.

However, pure SSMs exhibit consistent weaknesses on tasks requiring strong in-context learning or copying from context. NVIDIA research (2024) found that while Mamba and Mamba-2 match transformers on many benchmarks at 8B scale, they lag on five-shot MMLU and phonebook lookup tasks. This has driven increasing adoption of hybrid architectures: AI21's Jamba 1.5 Large scored 65.4 on Arena Hard, outperforming Llama-3.1-70B and 405B, using a 43% Mamba-2, 7% attention, 50% MLP architecture.

Estimated probability of pure SSMs being dominant at transformative AI: 5-15%. Probability of SSM-transformer hybrids playing significant role: 25-40%.

Architecture Comparison

The fundamental difference between transformers and SSMs lies in how they handle sequence dependencies. Transformers compute pairwise relationships between all tokens (quadratic), while SSMs compress history into a fixed-size state that evolves with each new token (linear).

The selection mechanism is Mamba's key innovation. Unlike prior SSMs where state dynamics (A, B, C matrices) were fixed, Mamba makes them input-dependent. This allows the model to:

• Remember important tokens by increasing their influence on state (large delta)
• Forget irrelevant tokens by letting state decay quickly (small delta)
• Focus on content-relevant patterns rather than just positional patterns

Key Differences

SSM Architecture Comparison

The SSM family has diversified rapidly since 2021. The following table compares major architectures:

Technical Details

Mamba Architecture

Mamba (Gu & Dao, 2023) introduced key innovations:

State-Space Formulation

h'(t) = Ah(t) + Bx(t)    # State evolution
y(t) = Ch(t) + Dx(t)     # Output

The key insight is that this continuous system can be discretized and computed efficiently using parallel scans. The matrices have interpretable roles: A (transition) controls how state information persists or decays, B (input) maps new tokens into state, C (output) maps state to predictions, and D provides skip connections. Mamba's innovation is making these parameters input-dependent (selective), allowing the model to decide what to remember or forget based on content.

Benchmark Performance Comparison

The following tables compile benchmark results from peer-reviewed papers comparing SSMs against transformers at similar scales.

Language Modeling Perplexity

Lower perplexity is better. Mamba achieves superior perplexity at equivalent scale.

Downstream Task Performance (8B Scale)

NVIDIA's empirical study (2024) provides the most comprehensive head-to-head comparison at production scale:

Hybrid architecture outperforms pure transformer by +1.3 points average while offering 8x faster inference.

Long Context Performance

SSMs excel at long context due to constant memory usage. RWKV-7 performance degrades rapidly beyond 28K.

Inference Efficiency

Mamba achieves 5x throughput and constant memory regardless of context length.

Key Properties

Safety Implications

The shift from attention to state-space dynamics has significant implications for AI safety research. SSMs present both opportunities and challenges that differ fundamentally from transformer-based systems.

Potential Safety Advantages

Safety Risks and Unknowns

Interpretability Comparison

The Gradient's analysis notes that while attention patterns in transformers provide intuitive visualizations of "what the model is looking at," SSM interpretability is fundamentally different:

"The precise selection mechanism's interpretability is less than that of attention visualizations, though selection weights can be probed."

Current Landscape

Production and Research Models (2024-2025)

Hybrid Architecture Design Patterns

The emergence of hybrid models reflects a growing consensus that pure SSMs and pure transformers each have fundamental limitations. Hybrids aim to capture the efficiency of SSMs with the in-context learning strength of attention.

NVIDIA's empirical study found the 43% SSM + 7% attention + 50% MLP configuration optimal at 8B scale, outperforming pure transformers by +2.65 points average while projecting 8x faster generation.

Research Landscape

Foundational Papers

Key Researchers and Organizations

Capability Assessment

Where SSMs Excel

Where Transformers Still Lead

Trajectory and Future Outlook

Quantified Adoption Drivers

Arguments for SSM/Hybrid Growth (60-70% probability of significant adoption)

1. Efficiency becomes critical — At GPT-5+ scale, O(n^2) attention cost is $10-100M per training run. SSM efficiency offers 40-80% cost reduction.
2. Long context is table stakes — Applications demand 100K-1M token contexts. Transformer KV cache hits memory limits; SSM scales linearly.
3. Hybrid architectures validated — NVIDIA's study and Jamba 1.5 demonstrate hybrids can outperform pure transformers with better efficiency.
4. Production deployments expanding — Google (RecurrentGemma), AI21 (Jamba 1.6), Mistral (Codestral Mamba), IBM (Granite 4.0) all shipping SSM-based models.

Arguments Against (30-40% probability SSMs remain niche)

1. In-context learning ceiling — Pure SSMs consistently underperform on MMLU, few-shot tasks. May be fundamental limit of stateful compression.
2. Transformer ecosystem lock-in — PyTorch, TensorFlow, vLLM, TensorRT all optimized for attention. Switching costs are substantial.
3. Investment momentum — >95% of frontier training compute goes to transformers. Network effects favor incumbents.
4. Interpretability gap — Safety teams trained on attention analysis. SSM interpretability tools 3-5 years behind.

Scenario Probabilities

Safety Research Implications

Research That Likely Transfers

• RLHF - Training approach similar
• Behavioral evals - Testing works the same
• Red teaming - Adversarial testing still applies

Research That May Not Transfer

• Attention-based interpretability - No attention to analyze
• Transformer-specific probes - Need new tools
• Circuit analysis - Different computational structure

Unique Research Opportunities

Critical Research Questions

The Fundamental Crux

The core uncertainty is whether the in-context learning limitation of pure SSMs is:

A. Fundamental — State compression inherently loses precise retrieval capability. Transformers' O(n) KV cache stores exact tokens; SSMs' O(1) state must compress. If true, hybrids will dominate.

B. Solvable — Better selection mechanisms, larger state dimensions, or architectural innovations could match transformer in-context learning. If true, pure SSMs could dominate due to efficiency.

Current evidence favors interpretation (A): NVIDIA's empirical study found that even at 8B scale with extensive training, pure Mamba-2 lags on MMLU (46.3% vs 51.2%) and phonebook lookup tasks. The 43% SSM + 7% attention hybrid closes this gap completely, suggesting attention provides irreplaceable retrieval capability.

Sources & Key References

Foundational Papers

• S4 (2021): Gu, A., Goel, K., & Ré, C. "Efficiently Modeling Long Sequences with Structured State Spaces". ICLR 2022.
• Mamba (2023): Gu, A. & Dao, T. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". ICLR 2024.
• Mamba-2 (2024): Dao, T. & Gu, A. "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality". ICML 2024.

Benchmark Studies

• NVIDIA Empirical Study: Waleffe, R. et al. "An Empirical Study of Mamba-based Language Models". 2024. Definitive 8B-scale comparison.
• Mamba-360 Survey: "Mamba-360: Survey of State Space Models as Transformer Alternative". Engineering Applications of AI, 2025.
• Comprehensive Survey: "From S4 to Mamba: A Comprehensive Survey on Structured State Space Models". arXiv, 2025.

Production Models

• Jamba: AI21 Labs. "Introducing Jamba: AI21's Groundbreaking SSM-Transformer Model". 2024.
• Jamba 1.5: AI21 Labs. "The Jamba 1.5 Open Model Family". 2024.
• RecurrentGemma: Google DeepMind. "RecurrentGemma Model Card". 2024.
• StripedHyena: Together AI. "StripedHyena-7B: Open Source Models Beyond Transformers". 2023.

Alternative Architectures

• Hyena: Poli, M. et al. "Hyena Hierarchy: Towards Larger Convolutional Language Models". ICML 2023.
• RWKV: Peng, B. et al. "RWKV: Reinventing RNNs for the Transformer Era". EMNLP 2023.
• Griffin: De, S. et al. "Griffin: Mixing Gated Linear Recurrences with Local Attention". ICML 2024.

Interpretability and Safety

• Mamba Explained: The Gradient. "Mamba Explained". 2024. Includes interpretability analysis.
• IBM Overview: IBM. "What Is A Mamba Model?". 2024.
• Visual Guide: Grootendorst, M. "A Visual Guide to Mamba and State Space Models". 2024.

Code and Implementations

• Official Mamba: github.com/state-spaces/mamba - Reference implementation by Gu & Dao.
• RWKV: github.com/BlinkDL/RWKV-LM - Community-driven RNN alternative.
• Hazy Research Blog: hazyresearch.stanford.edu - Stanford's SSM research hub.