Research map / Architectures and efficiency / Attention and transformers

Gated Delta Networks: Improving Mamba2 with Delta Rule

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the gated delta rule, combining Mamba2-style gated decay with the DeltaNet delta update, plus a hardware-efficient chunkwise training algorithm and hybrid variants with sliding window attention and Mamba2.

Problem

Linear Transformers and RNNs underperform in in-context retrieval and long-context tasks because their fixed-size memory suffers from collisions; Mamba2 decays all associations uniformly while DeltaNet lacks a fast memory-clearing mechanism.

Approach

Unify data-dependent gating (Mamba2) and the delta update rule (DeltaNet) into one recurrence S_t = S_{t-1}(alpha_t(I - beta_t k k^T)) + beta_t v k^T, enabling rapid memory erasure via the gate and targeted key-value updates via the delta rule, with an extended WY/UT chunkwise algorithm for hardware-efficient training.

Key results

• Gated DeltaNet 1.3B reaches 55.32 average zero-shot commonsense accuracy, above Mamba2 (54.89) and DeltaNet (52.14); hybrid Gated DeltaNet-H1 reaches 56.40.
• On S-NIAH-2 at 8K, Gated DeltaNet scores 29.6 vs Mamba2 17.0 and DeltaNet 14.4 (1.3B models), though it is below DeltaNet on S-NIAH-1 at 4K-8K.
• On LongBench 14-task average, Gated DeltaNet achieves 16.6 vs Mamba2 13.5 and DeltaNet 13.6; Gated DeltaNet-H2 reaches 18.4.

Measured effects reported in the paper

Surprising findings

• Pure DeltaNet beats Gated DeltaNet on S-NIAH-1 passkey retrieval at 4K-8K lengths, and Mamba2 beats DeltaNet on real-world recall tasks, showing synthetic retrieval gains do not always transfer.
• Gated DeltaNet's real-world retrieval margin over Mamba2 is smaller than on synthetic tasks, attributed to repetition errors of small non-instruction-tuned models.

Limitations

• Length extrapolation results are mixed, and performance on sequences beyond 20K tokens is untested.
• Real-world retrieval gains are modest because small non-instruction-tuned models suffer repetition errors.
• Experiments are limited to models at 400M-1.3B scale trained on 100B tokens.

Experimental setup

Datasets: FineWeb-Edu RULER S-NIAH SWDE FDA SQuAD TriviaQA DROP Natural Questions LongBench PIQA HellaSwag WinoGrande

Benchmarks: WikiText LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c SIQA BoolQ RULER S-NIAH-1/2/3 SWDE FDA

Metrics: perplexity accuracy throughput (tokens/sec)

Baselines: RetNet HGRN2 Mamba Mamba2 DeltaNet Samba Transformer++

Builds on: DeltaNet Mamba2 chunkwise parallel form of linear attention Flash Linear Attention

Closest papers by meaning

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.90)
• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.88)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.86)
• On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.86)
• When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning (ICML 2026 · similarity 0.85)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.85)