Research map / Architectures and efficiency / Attention and transformers
How Can Mamba Learn In Context with Outliers and Generalize Provably?
Hongkang Li, Songtao Lu, Xiaodong Cui, Pin-Yu Chen, Meng Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add nonlinear gating (Mamba instead of linear attention) (ablation) | one-layer linear Transformer · Synthetic ICL, alpha>1/2 | classification error: smaller than 0.01 | Improved |
| add nonlinear gating (Mamba instead of linear attention) (ablation) | three-layer linear attention · Synthetic ICL (d=72, alpha=0.5), FQ | test accuracy: 93.68 -> 99.73 | Improved |
| place outliers closest to the query instead of random | Mamba · Synthetic ICL (d=72, alpha=0.5) | test accuracy: 99.67 -> 82.73 | Hurt |
| place outliers closest to the query instead of random | linear attention Transformer · Synthetic ICL (d=72, alpha=0.5) | test accuracy: 94.12 -> 93.96 | No clear effect |
| add CQ data in second half of training (ablation) | Mamba · Synthetic ICL, CQ placement | ICL accuracy: 82.73 -> 95.45 | Improved |
| use 2 heads instead of 1 (ablation) | linear Transformer · Synthetic ICL, R placement | test accuracy: 94.12 -> 95.08 | Improved |
| use 4 heads instead of 1 (ablation) | linear Transformer · Synthetic ICL, R placement | test accuracy: 94.12 -> 90.90 | Hurt |
| replace linear attention with softmax attention (ablation) | three-layer Transformer · Synthetic ICL, CQ placement | test accuracy: 93.96 -> 99.28 | Improved |
| raise outlier fraction alpha from 0.4 to 0.7 | linear Transformer · Synthetic ICL, R placement | test accuracy: 94.12 -> 67.78 | Hurt |
| raise outlier fraction alpha from 0.4 to 0.7 | softmax Transformer · Synthetic ICL, R placement | test accuracy: 99.26 -> 95.58 | Hurt |
| keep 10 principal components instead of 5 | DistilBERT sentence vectors classifier · SST-2 PCA | accuracy: 74.08 -> 78.90 | Improved |
| use Mamba instead of linear Transformer with outlier phrase 'James Bond' (ablation) | 3-layer 2-head model · SST-2 ICL (alpha=0.75, FQ and R placement) | ICL accuracy | Improved |
Closest papers by meaning
- Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning (NeurIPS 2026 · similarity 0.92)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.91)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.89)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.88)
- Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression (NeurIPS 2025 · similarity 0.88)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
