Research map / Architectures and efficiency / Attention and transformers
DeciMamba: Exploring the Length Extrapolation Potential of Mamba
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows Mamba's length extrapolation is limited by an effective receptive field tied to training length, and proposes DeciMamba, a training-free context-extension method that pools tokens inside S6 layers using the Δ_t gate norms.
Problem
Mamba models, despite sub-quadratic cost, generalize poorly to sequences longer than those seen in training because their effective receptive field (ERF) collapses beyond the training length.
Approach
Use the per-token selective step size Δ_t as an importance score and keep only the top-P tokens before the S6 layers in deeper "global" layers (pre-fill only), so each decimated layer operates on sequence lengths similar to those seen in training; this extends context without retraining.
Key results
- DeciMamba-130M extends passkey retrieval success from about 16K (Mamba) to 128K when trained on 2K sequences.
- On LongBench, DeciMamba raises the instruction-tuned Mamba-2.8B LongBench average on GovReport from 9.8 to 14.9 and TriviaQA from 3.93 to 12.61.
- DeciMamba is about 2x faster than Mamba at 512K context in the inference benchmark (8.17s vs 4.22s for 130m model at 512K).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply delta-based token decimation | Mamba-2.8B Zephyr · TriviaQA (LongBench, LB) | F1: 3.93 -> 12.61 | Helped |
| Apply delta-based token decimation | Mamba-130M · Multi-document retrieval, 10 documents | accuracy: 68.3 -> 67.7 | Hurt |
| Apply delta-based token decimation | Mamba-130M · Multi-document retrieval, 100 documents | accuracy: 5.7 -> 37.0 | Helped |
| Apply delta-based token decimation | Mamba-130M · Multi-document QA, 100 documents | F1: 7.5 -> 18.1 | Helped |
| Apply delta-based token decimation | Mamba-2.8B · PG-19 test, 80K context | perplexity: inf -> 27.43 | Helped |
| Apply delta-based token decimation | Mamba-130M · Passkey Retrieval, 128K | success rate: ✗ -> ✓ | Helped |
| Apply delta-based token decimation | Mamba-130M · Passkey Retrieval, inference at 512K | inference time (seconds): 8.17 -> 4.22 | Helped |
| Replace Top-Ps pooling with Top-K% pooling (ablation) | DeciMamba-130M · Passkey Retrieval, extrapolation | maximum extrapolation length: 128K vs 32K | Hurt |
| Use delta-based pooling instead of max-norm pooling (ablation) | Mamba-1.4B · PG-19 zero-shot | perplexity | Helped |
| Use random pooling instead of processing the full sequence (ablation) | Mamba-1.4B · PG-19 zero-shot, longer contexts | perplexity | Helped |
| Add decimation during decoding (ablation) | DeciMamba · GovReport (LongBench, LB) | Rouge-L: 14.9 -> 14.7 | No clear effect |
| Set beta to 0.75 rather than 1 (ablation) | DeciMamba-130M · Passkey Retrieval | success rate: 1.0 -> 0.975 | No clear effect |
Surprising findings
- Random token pooling achieves better perplexity than no pooling at longer contexts, showing how strongly Mamba is hurt by limited ERFs.
- Adding decimation during decoding yields negligible change on GovReport, so pooling is only needed in pre-fill.
Limitations
- Modifies a pretrained Mamba without fixing the underlying architectural ERF issue; pooling may discard critical information.
- Performance still degrades at longer contexts in zero-shot LongBench settings.
- Experiments on multi-document QA use small 130M models with modest absolute scores.
Experimental setup
Datasets: WikiText-103 SQuAD v2 PG-19 Pile LongBench
Benchmarks: LongBench Passkey Retrieval Multi-Document Retrieval and QA (SQuAD v2-based) PG-19 perplexity
Metrics: accuracy F1 Rouge-L perplexity inference time
Baselines: Mamba Pythia max-norm pooling random pooling Top-K% pooling
Builds on: Mamba S6 selective state-space layers Landmark Attention LongLoRA Vision Transformer attention mean distance
Closest papers by meaning
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.90)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.89)
- Exploring Transformer Extrapolation (AAAI 2024 · similarity 0.89)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.88)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.88)
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.