Research map / Architectures and efficiency / RNNs and normalization
Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how input selectivity in Mamba's S6 layer affects function approximation (Haar wavelets), memory decay, and associative recall (MQAR, Induction Heads), with analytical 1-layer constructions and tightness experiments.
Problem
It is unclear how Mamba leverages input selectivity in its S6 SSM layer, and how this interacts with convolution and gating, for approximation power, long-term memorization, and associative recall.
Approach
Shows S6 can represent Haar wavelet projections via input-dependent discretization (beating S4D's Fourier bases), that S6 can counteract exponential sensitivity decay by adjusting Delta, and gives 1-layer Mamba/Mamba-2/Mamba-S4D constructions for MQAR with model size independent of sequence length, plus a Mamba-Delta^T variant for Induction Heads.
Key results
- Keep Fifth task: Mamba+PE reaches 1.00 accuracy matching Transformer, while Mamba without PE (0.08) and S4D (0.09) fail.
- Empirical MQAR thresholds across embedding sizes track theoretical bounds: Mamba-2 needs smaller model size than Mamba, which needs smaller than S4D.
- Mamba-Delta^T performs equal or better than Mamba on Induction Heads across model sizes and reaches 100% only above the theoretical bound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add positional encoding (ablation) | Mamba · Keep Fifth | accuracy: 0.08 -> 1.00 | Helped |
| add positional encoding (ablation) | S4D · Keep Fifth | accuracy: 0.09 -> 0.08 | Hurt |
| use Mamba-$\Delta^\top$ instead of standard Mamba, with selectivity along the state dimension (ablation) | Mamba · Induction Heads | accuracy | Helped |
Surprising findings
- A 1-layer Mamba with S4D mixer (no input selectivity in SSM) can still solve MQAR exactly via convolution and gating.
- Mamba without positional encoding fails the Keep Fifth task, unlike its success with PE.
Limitations
- Theory does not address optimization or generalization.
- Analysis restricted to simple synthetic associative recall tasks such as MQAR and Induction Heads.
- Constructions show possibility of solutions, not that trained models find them.
Experimental setup
Datasets: MQAR synthetic Induction Heads synthetic Keep n-th synthetic
Benchmarks: MQAR Induction Heads Keep Fifth
Metrics: accuracy parameter count
Baselines: S4D Mamba Mamba-2 Transformer
Builds on: Mamba S4D HiPPO Mamba-2 Multiple-Query Associative Recall
Closest papers by meaning
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.90)
- On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing (NeurIPS 2026 · similarity 0.88)
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.88)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.88)
- Mamba-Adaptor: State Space Model Adaptor for Visual Recognition (CVPR 2025 · similarity 0.88)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.