Research map / Architectures and efficiency / Attention and transformers
Block-Biased Mamba for Long-Range Sequence Processing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes Mamba's S6 limitations (expressiveness, inductive bias, training stability) versus S4D and proposes B2S6, a block-wise selective S6 variant with channel-specific input-independent bias, which tops LRA among selective SSMs.
Problem
Mamba (S6) performs poorly on long-range sequence tasks such as the Long-Range Arena, despite being built for long-range dependencies; the paper seeks to explain why and provide a remedy.
Approach
Partition the input into h blocks, each with its own selective S6-style recurrence, and add a channel-specific input-independent bias to the input matrix B, with reduced learning rate on Delta-related parameters; theory shows this restores universal approximation and gentler input-magnitude bias.
Key results
- B2S6 achieves 87.71% average accuracy on Long-Range Arena, versus 86.09% for S4 and 66.59% for S6 (Mamba).
- On sCIFAR-10 with d=128, B2S6 with bias reaches up to 87.19% accuracy (h=8, p=16), versus 71.77% without bias at h=1.
- B2S6 reportedly matches Mamba perplexity on sampled SlimPajama language modeling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use B₂S₆ instead of S5 | LRA · ListOps | accuracy: 63.77 -> 63.85 | Helped |
| Use B₂S₆ instead of S5 | LRA · Text | accuracy: 89.31 -> 88.32 | Hurt |
| Use B₂S₆ instead of S7 | LRA · Retrieval | accuracy: 91.80 -> 91.44 | Hurt |
| Use B₂S₆ instead of S4 | LRA · Image | accuracy: 88.65 -> 88.81 | Helped |
| Use B₂S₆ instead of S5 | LRA · Pathfinder | accuracy: 95.33 -> 95.93 | Helped |
| Use B₂S₆ instead of S5 | LRA · Path-X | accuracy: 98.58 -> 97.90 | Hurt |
| Use B₂S₆ instead of S5 | LRA overall · LRA average | accuracy: 87.46 -> 87.71 | Helped |
| Add channel-specific B_bias (ablation) | B₂S₆, sCIFAR-10 (h=1, p=128) · sCIFAR-10 | accuracy: 71.77 -> 81.56 | Helped |
| Add channel-specific B_bias (ablation) | B₂S₆, sCIFAR-10 (h=2, p=64) · sCIFAR-10 | accuracy: 79.44 -> 84.83 | Helped |
| Use complex instead of real parameters (ablation) | B₂S₆, sCIFAR-10 (h=1, p=128) · sCIFAR-10 | accuracy: 47.82 -> 81.56 | Helped |
| Use complex instead of real parameters (ablation) | B₂S₆, sCIFAR-10 (h=2, p=64) · sCIFAR-10 | accuracy: 54.40 -> 84.83 | Helped |
| Increase block count h from 1 to 8 with d fixed and no bias (ablation) | B₂S₆ · sCIFAR-10 | accuracy: 71.77 -> 81.54 | Helped |
Surprising findings
- Tying B and C across channels in S4D drops grayscale sCIFAR-10 accuracy from about 88% to below 80%.
- Real-parameter B2S6 with bias performs far worse (around 50%) on sCIFAR-10 than complex-parameter variants.
Limitations
- Training stability is only addressed by reducing learning rates; the theory fixes w=0 and the analysis is for single-layer models.
- Language modeling evaluation is preliminary and not at LLM scale.
- Block structure was previously proposed in Mamba2, so novelty is mainly the bias term and analysis.
Experimental setup
Datasets: Long-Range Arena sCIFAR-10 SlimPajama-627B (sampled)
Benchmarks: Long-Range Arena sCIFAR-10 SlimPajama
Metrics: test accuracy average accuracy perplexity relative error
Baselines: S4 S4D S5 S6 (Mamba) S7
Builds on: Mamba S4D Mamba2 S5 Long-Range Arena
Closest papers by meaning
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data (NeurIPS 2025 · similarity 0.91)
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.91)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.91)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.90)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.89)
- MambaLRP: Explaining Selective State Space Sequence Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.