Research map / Architectures and efficiency / Attention and transformers
Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hardware-efficient chunkwise parallel algorithm for training DeltaNet (linear transformers with the delta rule) across sequence length, using the WY/UT representation of Householder products, and scales it to 1.3B and 3B language models plus hybrid attention variants.
Problem
Linear transformers with the delta rule (DeltaNet) have better associative recall than additive linear attention, but prior training algorithms were sequential over sequence length and hardware-inefficient, so they could not be scaled to standard language modeling settings.
Approach
Reparameterize DeltaNet's recurrence as a generalized Householder update and use the WY/UT representation to express the state transitions in a memory-efficient form, enabling a chunkwise parallel algorithm that uses matmuls and tensor cores while only materializing the state at chunk boundaries.
Key results
- The 1.3B DeltaNet (w. conv) trained on 100B tokens reaches 16.87 Wikitext perplexity and 51.6 average zero-shot accuracy, beating Mamba (17.06 ppl, 50.0 avg) and GLA (17.22 ppl, 51.0 avg).
- Hybrid DeltaNet with global attention on two layers reaches 71.0 SWDE accuracy at 1.3B, above Transformer++ at 66.6.
- DeltaNet solves MQAR perfectly in the hardest setting even without convolution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the delta-rule update instead of GLA's gated linear update | Linear recurrent language models · LAMBADA, 1.3B params / 100B tokens | perplexity: 14.47 -> 12.21 | Helped |
| Use the delta-rule update instead of GLA's gated linear update | Linear recurrent language models · Zero-shot average, 1.3B params / 100B tokens | accuracy: 51.0 -> 51.6 | Helped |
| Use the delta-rule update instead of GLA's gated linear update | Linear recurrent language models · SWDE, SQuAD, FDA, 1.3B params / 100B tokens | accuracy: 52.4 -> 49.5; 42.6 -> 37.4; 22.3 -> 17.2 | Hurt |
| Interleave sliding-window attention every other layer | DeltaNet · Zero-shot average, 1.3B params / 100B tokens | accuracy: 51.6 -> 52.1 | Helped |
| Replace two layers with global attention | DeltaNet · SWDE, 1.3B params / 100B tokens | accuracy: 49.5 -> 71.0 | Helped |
| Use L2 normalization instead of L1 normalization, with 1+ELU feature map (ablation) | DeltaNet · Zero-shot average, 340M params / 15B tokens | accuracy: 40.1 -> 42.1 | Helped |
| Use SiLU instead of 1+ELU with L2 normalization (ablation) | DeltaNet · Zero-shot average, 340M params / 15B tokens | accuracy: 42.1 -> 42.1 | Helped |
| Use ReLU instead of 1+ELU with L2 normalization (ablation) | DeltaNet · Zero-shot average, 340M params / 15B tokens | accuracy: 42.1 -> 40.9 | Hurt |
Surprising findings
- At the 1.3B scale DeltaNet underperforms GLA on recall-intensive tasks (SWDE, FDA), attributed to smaller recurrent state size.
- DeltaNet shows limited length generalization beyond the training length, unlike GLA, RetNet, and Mamba.
Limitations
- Training speed still lags behind GLA because of state-to-state dependencies that require marginalizing over the head dimension.
- Limited length generalization since DeltaNet lacks explicit decay factors.
- Recurrent state size is limited by head dimension, hurting recall-intensive performance at larger scale.
Experimental setup
Datasets: SlimPajama MQAR MAD RegBench Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-easy ARC-challenge FDA
Benchmarks: Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-easy ARC-challenge FDA SWDE SQuAD MQAR MAD
Metrics: perplexity accuracy normalized accuracy training throughput
Baselines: Transformer++ RetNet Mamba GLA Hyena Llama-3.2-3B PowerLM-3B
Builds on: DeltaNet chunkwise parallel linear attention FlashLinearAttention WY representation Gated Linear Attention
Closest papers by meaning
- Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025 · similarity 0.90)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.89)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.88)
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.88)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.