Research map / Architectures and efficiency / Attention and transformers
Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows SSMs (Mamba, S4, linear attention) have inherent recency bias and over-smoothing with depth, and proposes polarizing two state-transition channels (one fixed to 1, one fixed to 0) to mitigate both, improving associative recall.
Problem
Structured state space models are often claimed to capture long-range dependencies, but they exhibit strong recency bias and, as depth grows, over-smoothing of token representations, which limits long-context recall and scalability.
Approach
Theoretically proves exponential decay of influence with token distance and a smoothing (over-smoothing) bound for SSM recurrences, then proposes polarization: fix one state-transition channel to 1 (preserving history) and another to 0 (focusing on current token), with the rest learnable.
Key results
- Polarizing both channels in a 4-layer Mamba raises average associative recall accuracy from 71.61 to 93.43 across 64/128/256 KV pairs.
- On CIFAR-10 with 1024-token sequences, corrupting the last 32 tokens drops Mamba accuracy by 81.24% versus 2.30% for the first 32 tokens.
- Over 60% of channels have A_max - A_min below 0.5, indicating SSMs cannot simultaneously avoid recency and over-smoothing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Corrupt the last 32 tokens rather than the first 32 | Mamba · CIFAR-10, sequences of 1,024 pixel tokens | classification accuracy: 0.674 -> 0.126; 0.674 -> 0.658 | Hurt |
| Increase depth from 2 to 4 layers without polarization (ablation) | Mamba · Associative recall, 256 KV pairs | accuracy: 36.00 -> 33.52 | Hurt |
| Polarize one state channel to one (ablation) | Mamba, 2 layers · Associative recall, average | accuracy: 72.06 -> 83.63 | Helped |
| Polarize one state channel to zero (ablation) | Mamba, 2 layers · Associative recall, average | accuracy: 72.06 -> 72.10 | No clear effect |
| Polarize one state channel to zero and increase depth from 2 to 4 layers (ablation) | Mamba · Associative recall, average | accuracy: 72.06 -> 81.38 | Helped |
| Polarize one state channel to one and another to zero (ablation) | Mamba, 2 layers · Associative recall, average | accuracy: 72.06 -> 83.17 | Helped |
| Polarize one state channel to one and another to zero, and increase depth from 2 to 4 layers (ablation) | Mamba · Associative recall, average | accuracy: 72.06 -> 93.43 | Helped |
Surprising findings
- Mamba with default parameterization performs worse as depth increases in associative recall (71.61 at 4 layers vs 72.06 at 2 layers).
- Deeper Mamba models (over 48 layers) at 2048 context underperform 16-layer models in pretraining validation loss.
Limitations
- Theoretical results assume non-expansive recurrences and (for some bounds) positivity or centering assumptions on inputs.
- Empirical scaling and polarization experiments are on small-scale settings; the 7B Needle test is only diagnostic and polarization is not evaluated at large scale.
Experimental setup
Datasets: CIFAR-10 associative recall key-value data language modeling pretraining corpus
Benchmarks: Needle in a Haystack CIFAR-10 adversarial and target attack associative recall
Metrics: retrieval accuracy classification accuracy attack success rate validation perplexity influential score feature smoothness
Baselines: Mamba H3 RWKV Transformer Mistral-7B Pythia
Builds on: Mamba S4 HiPPO H3 Linear attention
Closest papers by meaning
- Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.88)
- Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.88)
- The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.88)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.87)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.87)
- Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.