Research map / Large language models / Reasoning and chain of thought
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales the xLSTM (mLSTM-only) architecture to a 7B-parameter LLM trained on 2.3T tokens, with a post-up-projection block, gate soft-capping and negative input-gate bias init, and reports fast generation/prefill throughput versus Llama and Mamba models.
Problem
Recurrent LLMs with linear-time, constant-memory inference have not been scaled beyond about 1.3B parameters, and their inference speed and training stability at 7B scale were unknown.
Approach
Replace the previous xLSTM pre-up-projection block (mLSTM with convolutions and skip connections, operating in a higher dimension) with a Transformer-style post-up-projection block (mLSTM in embedding dimension followed by SwiGLU MLP), and stabilize training via RMSNorm pre-norm, soft-capping of input/forget gates, and negative input-gate bias initialization.
Key results
- xLSTM 7B reaches an average of 0.260 on Open LLM Leaderboard v2, mid-range among 7B models, and 0.613 average on v1 tasks.
- On the speed benchmarks, xLSTM 7B is about 50% faster than Mamba in text generation and its prefill throughput is about 70% higher than Codestral Mamba.
- Per-block redesign gives a 2.64x to 3.64x training throughput speedup over the previous xLSTM architecture across 160M to 7B sizes, at a small perplexity cost at 1.4B (+0.22 ppl).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the pre-up projection block with the post-up projection block (ablation) | xLSTM · 1.4B model training | throughput: 10.57 -> 37.03; 3.5× speedup | Helped |
| Replace the pre-up projection block with the post-up projection block (ablation) | xLSTM · 1.4B model validation | perplexity: 12.46 -> 12.68; +0.22 | Hurt |
| Use post-up projection block instead of the previous block (ablation) | xLSTM · 7B model training | throughput: 3.46 -> 9.15; × 2.64 | Helped |
| Increase the head count from 8 to 16 (ablation) | xLSTM 7B · 7B model validation | validation perplexity: 9.52 -> 9.52 | No clear effect |
| Increase the head count from 8 to 32 (ablation) | xLSTM 7B · 7B model training | train step time: 3.63 -> 3.41 | Helped |
| Use 4 heads instead of 8 (ablation) | xLSTM 7B · RULER long-context evaluation | average accuracy | Mixed |
| Replace the pre-norm LayerNorm with RMSNorm (ablation) | xLSTM · 1.4B model training | training stability | Helped |
| Replace the mLSTM state-norm LayerNorm with RMSNorm (ablation) | xLSTM · 1.4B model training | training stability | No clear effect |
| Apply soft-capping to the input and forget gate pre-activations and output logits (ablation) | xLSTM 7B · 160B-token training | validation loss | Helped |
| Initialize the input gate bias to -10 rather than higher values (ablation) | xLSTM · 160M model training | validation perplexity: -10 | Helped |
| Make the input gate non-trainable at 0 (ablation) | xLSTM 7B · Huggingface Leaderboard v1 | Average: 0.613 -> 0.510 | Hurt |
| Make the input gate non-trainable at -10 (ablation) | xLSTM 7B · RULER long-context evaluation | average accuracy | Hurt |
Limitations
- Downstream performance is only mid-range among 7B models and trails the strongest models such as Qwen2.5-7B, which the authors attribute to less math and code data.
- The 7B ablations were trained on only 160B tokens, so the memory-state-size effect on long context is inconclusive.
- Fixed-size memory may limit long-context recall compared to growing KV caches.
Experimental setup
Datasets: DCLM FineWeb-Edu Cosmopedia ProofPile-2 TheStack NuminaMath CoT MetaMathQA Tulu v3.1 OpenHermes 2.5 GSM8K Smoltalk LongDataCollections
Benchmarks: Open LLM Leaderboard v2 (BBH, MMLU-Pro, MATH, MuSR, GPQA, IFEval) Open LLM Leaderboard v1 tasks via lighteval RULER generation throughput time to first token prefill throughput
Metrics: accuracy tokens per second latency (time to first token) GPU memory validation perplexity
Baselines: Llama-2-7B Llama-3.1-8B Falcon-Mamba-7B Codestral Mamba RWKV-v5/6 Zamba2-7B Qwen2.5-7B Gemma-7B
Builds on: xLSTM Mamba LLaMA FlashLinearAttention
Closest papers by meaning
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.90)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.90)
- xLSTM: Extended Long Short-Term Memory (NeurIPS 2024 · similarity 0.90)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.90)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.90)
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.