Research map / Architectures and efficiency / Attention and transformers
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that selective SSMs (with scalar-identity A) and masked linear attention are duals via semiseparable matrices, and uses this to design the SSD block-decomposition algorithm and the Mamba-2 architecture, which is 2-8x faster than Mamba's scan.
Problem
Structured state space models (SSMs) like Mamba and attention variants have been developed separately, making SSMs harder to train efficiently and to analyze with the tools developed for Transformers; a unified theory and efficient algorithm are needed.
Approach
Connects SSMs to semiseparable matrices, so that SSM computation is a structured matrix multiplication; generalizes linear attention to structured masked attention (SMA) and shows that 1-semiseparable SMA with scalar-identity SSMs are duals (state space duality). A block decomposition (chunked) algorithm combines the linear recurrence across chunks with quadratic attention-style matmuls within chu
Key results
- The SSD algorithm is 2-8x faster than Mamba's selective scan implementation while allowing ~8x larger state sizes with minimal slowdown.
- SSD is faster than FlashAttention-2 beyond sequence length 2K and about 6x faster at sequence length 16K.
- Mamba-2 2.7B trained on 300B Pile tokens outperforms Mamba-2.8B, Pythia-2.8B and Pythia-6.9B on the same data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SSD block-decomposition algorithm instead of the optimized selective scan | SSM computation · SSM computation | speed: 2-8$\times$ faster | Helped |
| Use SSD instead of softmax attention | SSM computation · Sequence length 16K | speed: 6$\times$ faster | Helped |
| Use SSD instead of softmax attention | SSM computation · Sequence length 2K | speed: crossing over at sequence length 2K | Mixed |
| Increase recurrent state size | SSD · SSM computation | speed: $8\times$ the size of Mamba or even higher, with minimal slowdown | No clear effect |
| Use the modified parallel Mamba block with SSD as the inner SSM layer | Mamba-2 · Chinchilla scaling laws | perplexity and wall-clock time | Helped |
| Train Mamba-2 2.7B on the Pile for 300B tokens | Mamba-2 2.7B · The Pile, 300B tokens | downstream evaluations | Helped |
Surprising findings
- Any kernel attention method with a fast recurrent form must be an SSM, as the paper proves for autoregressive structured attention.
- Restricting A to scalar-times-identity slightly reduces expressivity but enables matmul-heavy training that is much faster than Mamba's scan.
Limitations
- SSD's scalar-identity A is less expressive than Mamba's diagonal selective SSM.
- Experiments are limited to models up to about 2.7B parameters trained on the Pile in this excerpt.
Experimental setup
Datasets: The Pile
Benchmarks: The Pile perplexity LAMBADA HellaSwag PIQA ARC-Easy ARC-Challenge WinoGrande OpenBookQA multi-query associative recall
Metrics: perplexity accuracy training wall-clock time kernel runtime
Baselines: Mamba Transformer++ Pythia FlashAttention-2 GPT-3
Builds on: Mamba linear attention S4 FlashAttention-2 RetNet
Closest papers by meaning
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.92)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.90)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.90)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.89)
- Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention (NeurIPS 2025 · similarity 0.89)
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.