Research map / Architectures and efficiency / Attention and transformers
Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses synthetic composite-function and inverse-sequence-matching tasks to show Mamba's nonlinear convolution induces an asymmetry bias that hinders symmetric solutions and reversed-sequence matching, and that a residual connection bypassing the convolution helps.
Problem
Mamba's architectural differences from Transformers (notably its nonlinear convolution before the SSM) are poorly understood, and it is unclear why Mamba struggles with symmetric patterns and reversed-sequence matching.
Approach
Attributes Mamba's failures on symmetric/reversed patterns to the asymmetric weights of the nonlinear convolution preceding the SSM, showing via blocking, substitution, and all-one-kernel ablations that the SSM is not the culprit; adding a residual path from convolution input to SSM plus positional encoding fixes inverse matching.
Key results
- Standard Mamba reaches 100% training accuracy on inverse sequence matching but random-guess test accuracy, while Transformer solves the task.
- Replacing convolution kernels with all-ones makes Mamba bias toward the symmetric solution on the composite function task, indicating convolution asymmetry drives the bias.
- A residual connection from convolution input to SSM plus positional encoding lets a 2-layer Mamba outperform the Transformer on inverse sequence matching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use standard Mamba with nonlinear convolution | Mamba · Composite function task | accuracy on symmetric solutions | Hurt |
| Set all convolution kernel weights to 1 (ablation) | Mamba · Composite function task | accuracy on symmetric solutions | Helped |
| Add positional encoding (ablation) | Mamba · Composite function task | accuracy on symmetric solutions | Helped |
| Block SSM information flow from the key and anchors to downstream tokens (ablation) | Mamba · Composite function task | output accuracy | No clear effect |
| Replace downstream post-convolution hidden states with states from a sequence using anchor pair 43 (ablation) | Mamba · Composite function task | output accuracy | Hurt |
| Use a Transformer with nonlinear convolution before attention (ablation) | Transformer · Composite function task | accuracy on symmetric solutions | Hurt |
| Use standard Mamba with nonlinear convolution | Mamba · Inverse sequence matching | test accuracy | Hurt |
| Use a Transformer instead of standard Mamba | Inverse sequence matching · Inverse sequence matching | test accuracy | Helped |
| Add a residual connection bypassing nonlinear convolution and positional embedding (ablation) | Mamba · Inverse sequence matching | test accuracy | Helped |
| Replace the residual bypass with a gating mechanism that combines raw and convolution-fused token information (ablation) | Mamba · Inverse sequence matching | accuracy | Mixed |
| Vary initialization rate from small to large initialization (ablation) | Mamba · Composite function task | accuracy on composite and symmetric solutions | Mixed |
Surprising findings
- Transformers with added convolution also show a composite-solution preference, similar to Mamba.
- Blocking SSM connections leaves Mamba's composite-task outputs nearly unchanged, suggesting the SSM plays little role in this task.
Limitations
- Experiments use small Mamba models and synthetic data, so results may not transfer to large-scale or real-world settings.
- Unclear whether the bias persists in hybrid Mamba-Transformer architectures.
- Error bars are not reported; results averaged over few seeds.
Experimental setup
Datasets: synthetic composite function data synthetic inverse sequence matching data
Benchmarks: synthetic composite function task synthetic inverse sequence matching task
Metrics: accuracy loss
Baselines: Transformer standard Mamba
Builds on: Mamba Mamba2 Transformer
Closest papers by meaning
- Block-Biased Mamba for Long-Range Sequence Processing (NeurIPS 2025 · similarity 0.91)
- Vision Mamba Mender (NeurIPS 2024 · similarity 0.90)
- Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity (ICML 2025 · similarity 0.90)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.89)
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML 2024 · similarity 0.89)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.