Research map / Architectures and efficiency / Attention and transformers
xLSTM: Extended Long Short-Term Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes xLSTM, extending LSTM with exponential gating and two new memory cells (scalar sLSTM with memory mixing, matrix-memory mLSTM), stacked into residual xLSTM blocks and scaled to 1.3B-parameter language models that beat Transformers, SSMs and RNNs on perplexity.
Problem
LSTMs were outpaced by Transformers at scale due to limited storage (scalar cell states), inability to revise storage decisions, and lack of parallelizability from memory mixing; the question is how far LSTMs go when scaled to LLM sizes.
Approach
Introduce exponential gating with normalizer and stabilizer states to let LSTMs revise storage decisions; sLSTM adds scalar memory with new head-wise memory mixing, mLSTM replaces the scalar cell with a matrix memory updated by a covariance rule and is fully parallelizable; both are wrapped in pre-LayerNorm residual blocks (post up-projection for sLSTM, pre up-projection for mLSTM).
Key results
- On 15B SlimPajama tokens, xLSTM[1:0] reaches 13.43 validation perplexity, beating Mamba (13.70) and Llama (14.25) at 350M size.
- At 1.3B on 300B tokens, xLSTM[1:0] reaches 8.89 SlimPajama perplexity vs Mamba 9.14 and Llama 9.44, with average downstream accuracy 58.48 vs 58.41 for Mamba.
- xLSTM[1:0] has lower PALOMA perplexity than Mamba on 568 of 571 text domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add exponential gating (ablation) | LSTM with Up-Projection Backbone · SlimPajama validation (15B tokens) | perplexity: 26.01 -> 17.70 | Helped |
| add matrix memory (ablation) | xLSTM[0:1] · SlimPajama validation (15B tokens) | perplexity: 17.70 -> 13.48 | Helped |
| add Resnet backbone (ablation) | vanilla LSTM · SlimPajama validation (15B tokens) | perplexity: 2417.86 -> 35.46 | Helped |
| add Up-Projection Backbone (ablation) | LSTM with Resnet Backbone · SlimPajama validation (15B tokens) | perplexity: 35.46 -> 26.01 | Helped |
| use input-dependent learnable forget and input gates (ablation) | xLSTM[1:0] · SlimPajama validation (15B tokens) | perplexity: 13.95 -> 13.43 | Helped |
| use input-dependent learnable forget gate only (ablation) | xLSTM[1:0] · SlimPajama validation (15B tokens) | perplexity: 13.95 -> 13.58 | Helped |
| use input-dependent learnable input gate only (ablation) | xLSTM[1:0] · SlimPajama validation (15B tokens) | perplexity: 13.95 -> 13.69 | Helped |
| use learnable forget- and input-gate biases without input-dependent gating (ablation) | xLSTM[1:0] · SlimPajama validation (15B tokens) | perplexity: 13.95 -> 13.73 | Helped |
| use input-dependent learnable gates rather than bias-only gates (ablation) | xLSTM[1:0] · SlimPajama validation (15B tokens) | perplexity: 13.73 -> 13.43 | Helped |
| use xLSTM rather than Mamba | 1.3B language models trained on SlimPajama · SlimPajama validation (300B tokens) | perplexity: 9.14 -> 8.89 | Helped |
| use xLSTM[1:0] rather than Mamba | 1.3B language models trained on SlimPajama · PALOMA average perplexity | perplexity: 11.84 -> 11.58 | Helped |
| use xLSTM[1:0] rather than xLSTM[7:1] | 1.3B language models trained on SlimPajama · PALOMA average perplexity | perplexity: 11.69 -> 11.58 | Helped |
Surprising findings
- Transformers and SSMs without memory mixing cannot solve regular-grammar tasks such as parity, while sLSTM with memory mixing can.
- Ablation shows a naive LSTM with exponential gating alone gains substantially; plain multi-layer LSTM reaches perplexity 2417 before backbone changes.
Limitations
- sLSTM memory mixing prevents full parallelization; its CUDA kernel is still under 2x slower than mLSTM.
- mLSTM CUDA kernels are unoptimized, about 4x slower than FlashAttention or Mamba scan.
- Hyperparameters and architecture were not fully optimized for larger xLSTM models due to compute cost.
Experimental setup
Datasets: SlimPajama Wikitext-103 PALOMA
Benchmarks: SlimPajama PALOMA LAMBADA HellaSwag PIQA ARC-E ARC-C WinoGrande Long Range Arena Multi-Query Associative Recall
Metrics: perplexity accuracy throughput generation time
Baselines: GPT-3 Llama Mamba RWKV-4 RWKV-5 RWKV-6 HGRN2 RetNet
Builds on: LSTM Fast Weight Programmers Mamba GLA
Closest papers by meaning
- xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.90)
- Vision-LSTM: xLSTM as Generic Vision Backbone (ICLR 2025 · similarity 0.88)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.88)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.87)
- UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (ICLR 2026 · similarity 0.87)
- On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.