Research map / Architectures and efficiency / Attention and transformers
LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongMamba, a training-free method that identifies global hidden-state channels in Mamba models and filters low-Δ tokens from their state updates to enlarge receptive fields beyond the training length.
Problem
Mamba/SSM language models underperform Transformers on long-context understanding and fail to generalize beyond their training sequence length because global hidden-state channels suffer exponential cumulative decay.
Approach
Classify hidden channels as local or global by cumulative decay over the training length, then for global channels skip state updates for tokens with small Δ_t (setting A'=1, B'=0), with a per-channel lookup table g(S) calibrated so the cumulative decay at length S matches that at training length L.
Key results
- On LongBench-E, LongMamba raises average accuracy from 8.37 to 17.33 on Mamba-1.4B (DeciMamba: 13.38), and by 8.76%, 9.13%, and 6.39% over vanilla Mamba-1.4B, Mamba2-1.3B, and Zamba2-1.2B.
- On RULER with Zamba2-1.2B at 32k tokens, LongMamba improves average accuracy from 2.72% to 13.83%, with S2 passkey going from 0% to 73%.
- On PG-19, LongMamba keeps perplexity below 20 up to 60k tokens across three models, while vanilla exceeds 40 beyond 10k tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply LongMamba instead of vanilla inference | Mamba-1.4B · LongBench-E | average accuracy (%): 8.37 -> 17.33 | Helped |
| apply LongMamba instead of DeciMamba | Mamba-1.4B · LongBench-E | average accuracy (%): 13.38 -> 17.33 | Helped |
| apply LongMamba instead of vanilla inference | Mamba2-1.3B · LongBench-E | average accuracy (%): 8.21 -> 17.34 | Helped |
| apply LongMamba instead of vanilla inference | Zamba2-1.2B · LongBench-E | average accuracy (%): 11.43 -> 17.82 | Helped |
| apply LongMamba instead of vanilla inference | Zamba2-1.2B · RULER 16k | average accuracy (%): 7.00 -> 31.58 | Helped |
| apply LongMamba instead of vanilla inference | Zamba2-1.2B · RULER 24k | average accuracy (%): 7.04 -> 22.22 | Helped |
| apply LongMamba instead of vanilla inference | Zamba2-1.2B · RULER 32k | average accuracy (%): 2.72 -> 13.83 | Helped |
| apply LongMamba instead of vanilla inference | Falcon-Mamba-7B · LongBench | average accuracy: 23.4 -> 26.2 | Helped |
| set channel selection threshold θ to 5.0×10^-2 rather than 7.5×10^-2 (ablation) | LongMamba on Mamba2-1.3B · LongBench-E | average accuracy (%): 16.81 -> 17.34 | Helped |
| vary channel selection threshold θ across 7.5×10^-2 to 7.5×10^-3 (ablation) | LongMamba on Mamba2-1.3B · LongBench-E | average accuracy (%): accuracy gap remains within 1.2% | No clear effect |
| set channel selection threshold θ to 1.0×10^-5 rather than 5.0×10^-2 (ablation) | LongMamba on Mamba2-1.3B · LongBench-E | average accuracy (%): 17.34 -> 14.87 | Hurt |
| vary the randomly sampled calibration sequence group (ablation) | LongMamba on Mamba2-1.3B · LongBench-E | average accuracy (%): STD 0.42% | No clear effect |
Surprising findings
- Increasing the threshold on channel selection barely changes accuracy (gap within 1.2% across a wide range), indicating robustness to the hyperparameter.
Limitations
- Relies on a threshold theta and clamping C% chosen by search on LongBench-E, which is also used for evaluation.
- Lookup table g(S) assumes input sequences of length S follow the same Delta distribution as calibration data.
- Evaluated on a limited set of Mamba-family models; Transformers remain stronger on several tasks.
Experimental setup
Datasets: Pile PG-19 RULER LongBench-E LongBench
Benchmarks: PG-19 RULER LongBench-E LongBench
Metrics: perplexity accuracy prefilling latency
Baselines: Vanilla Mamba-1.4B Vanilla Mamba2-1.3B Vanilla Zamba2-1.2B DeciMamba Llama2-7B-chat-4k XGen-7B-8k Vicuna-v1.5-7B-16k
Builds on: Mamba Mamba-2 DeciMamba
Closest papers by meaning
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.91)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.91)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.91)
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
- MambaLRP: Explaining Selective State Space Sequence Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.