Research map / Architectures and efficiency / Attention and transformers
Longhorn: State Space Models are Amortized Online Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Longhorn, a deep SSM derived from the closed-form implicit online regression update, which needs no separate forget gate and outperforms Mamba on MQAR, OpenWebText scaling, and 1.3B SlimPajama language modeling.
Problem
Sequence modeling with SSMs lacks a guiding design principle; linear recurrences are often ad hoc, and Transformers have quadratic cost for long contexts.
Approach
View the SSM recurrence as solving an online learning problem; choose an online associative-recall regression objective with a Frobenius-norm stability term, whose closed-form implicit update yields a diagonal-approximated state transition that automatically balances forgetting and learning without a separate forget gate.
Key results
- On SlimPajama 1.3B (100B tokens), Longhorn reaches a 52.35 average zero-shot score versus 52.25 for Mamba, with about 1.8x better sample efficiency than Mamba.
- Longhorn trained at 2048 context extrapolates up to 16x longer context (32K) with little perplexity increase, whereas DeltaNet does not extrapolate.
- On ImageNet, ViL-Small reaches 80.7% top-1 versus 80.5% for ViM-Small at 26M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Longhorn instead of Mamba for the SSM module | 1.3B language model trained on SlimPajama · SlimPajama downstream tasks | average perplexity during training: 1.8x | Helped |
| Use Longhorn instead of Mamba for the SSM module | 1.3B language model trained on SlimPajama · SlimPajama, eight downstream tasks | average score: 52.25 -> 52.35 | Helped |
| Use Longhorn instead of Mamba for the SSM module | OpenWebText language model · OpenWebText, 120M-scale, T=1024 | validation loss: 3.238 -> 3.225 | Helped |
| Use Longhorn instead of Mamba for the SSM module | OpenWebText language model · OpenWebText, 120M-scale, T=4096 | validation loss: 3.231 -> 3.192 | Helped |
| Use Longhorn instead of LLaMA | OpenWebText language model · OpenWebText, 350M-scale, T=1024 | validation loss: 2.891 -> 2.888 | Helped |
| Use Longhorn instead of Mamba | OpenWebText language model · OpenWebText, 350M-scale, T=4096 | validation loss: 2.868 -> 2.859 | Helped |
| Replace ViM with Vision Longhorn (ViL) | Vision SSM on ImageNet · ImageNet, Tiny configuration | top-1 accuracy: 76.1 -> 76.4 | Helped |
| Replace ViM with Vision Longhorn (ViL) | Vision SSM on ImageNet · ImageNet, Small configuration | top-1 accuracy: 80.5 -> 80.7 | Helped |
| Train Longhorn with context length 2048 and evaluate at longer context lengths | Longhorn pretrained on SlimPajama · SlimPajama length extrapolation | context length: 16x | Helped |
Surprising findings
- Longhorn remains competitive with Mamba on several downstream tasks despite having fewer parameters, since it needs no forget gate.
Limitations
- Uses a diagonal approximation of the closed-form update rather than the exact matrix form.
- Gains over Mamba on downstream tasks are small (about 0.1 average points).
Experimental setup
Datasets: SlimPajama OpenWebText ImageNet MQAR
Benchmarks: MQAR OpenWebText validation loss PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA SIQA BoolQ ImageNet
Metrics: recall rate validation loss perplexity zero-shot accuracy top-1 accuracy
Baselines: Mamba GLA RetNet RWKV LLaMA Based Hyena BaseConv Attention Vision Mamba (ViM) DeltaNet
Builds on: Mamba Gated Linear Attention DeltaNet Vision Mamba
Closest papers by meaning
- State Space Models are Provably Comparable to Transformers in Dynamic Token Selection (ICLR 2025 · similarity 0.89)
- LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement (ICLR 2025 · similarity 0.88)
- Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.88)
- Bridging Expressivity and Scalability with Adaptive Unitary SSMs (NeurIPS 2025 · similarity 0.87)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.87)
- The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.