Research map / Architectures and efficiency / Attention and transformers
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeltaProduct, a linear RNN whose state-transition matrices are products of n_h generalized Householder transformations (one online gradient step per key/value pair per token), improving state-tracking expressivity and length extrapolation over DeltaNet.
Problem
Linear RNNs face a trade-off between expressivity and efficiency set by the structure of their state-transition matrices; diagonal ones cannot solve group word problems like S_3 or S_5 in finite precision, while DeltaNet (diagonal plus rank-1) is limited to parity-like tasks in one layer and degrade
Approach
Take n_h gradient-descent steps per token on DeltaNet's associative recall loss using n_h keys, values and betas, so the state-transition matrix becomes a product of n_h generalized Householder matrices (identity plus rank-n_h), giving a tunable expressivity-efficiency knob with bounded spectral norm.
Key results
- DeltaProduct_2[-1,1] learns S_3 with one layer at length extrapolation beyond 128, where DeltaNet needs three layers; S_5 needs n_h=4 in a single layer.
- DeltaProduct_2[-1,1] with 8 heads (392M params) matches DeltaNet with 12 heads while achieving significantly better length extrapolation.
- Increasing n_h improves length extrapolation sharply from 1 to 2 Householders, with minimal degradation at n_h=3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase Householder products per token to n_h=2 (ablation) | Single-layer DeltaProduct[-1,1] · S3 state tracking | accuracy / length extrapolation | Helped |
| Increase Householder products per token to n_h=4 (ablation) | Single-layer DeltaProduct[-1,1] · S5 state tracking | accuracy / length extrapolation | Helped |
| Increase Householder products per token from n_h=1 to n_h=2 (ablation) | Single-layer DeltaProduct[-1,1] · S4 and A5 state tracking | accuracy / length extrapolation | Helped |
| Restrict the state-transition eigenvalue range to [0,1] instead of [-1,1] (ablation) | DeltaProduct · Permutation-group state tracking | accuracy | Hurt |
| Add layers while keeping n_h=1 (ablation) | DeltaProduct / DeltaNet[-1,1] · Permutation-group state tracking | accuracy | Helped |
| Use n_h=1 with additional layers instead of increasing n_h (ablation) | DeltaProduct / DeltaNet[-1,1] · Permutation-group state tracking | length extrapolation | Hurt |
| Add a forget gate (ablation) | DeltaProduct_2[-1,1] · Language-model length extrapolation | cross-entropy loss | Helped |
| Increase Householder products per token from one to multiple (ablation) | DeltaProduct[-1,1] · CodeParrot, OpenThoughts-114k-Math, and TriviaQA length extrapolation | cross-entropy loss | Helped |
| Increase Householder products per token to n_h=3 (ablation) | DeltaProduct[-1,1] · Language-model length extrapolation | cross-entropy loss | Helped |
| Increase n_h rather than model capacity through head dimension (ablation) | DeltaProduct versus DeltaNet · FineWeb | training perplexity | Helped |
| Increase n_h rather than model capacity through head dimension (ablation) | DeltaProduct versus DeltaNet · lm-eval-harness tasks | benchmark performance | Helped |
| Increase n_h (ablation) | DeltaProduct and Gated DeltaProduct · Language-modeling benchmarks | benchmark performance | Helped |
Surprising findings
- S_4 and A_5 extrapolate robustly with only n_h=2, below the n_h suggested by the general construction, because they embed in SO(3) and the learned model uses reflections with beta near 2 and keys in a 3D subspace.
- DeltaProduct_3[-1,1] without a forget gate achieves performance comparable to Gated DeltaNet[-1,1].
Limitations
- Training compute and recurrence cost scale linearly with n_h.
- Extra key/value projections add parameters with higher n_h, requiring parameter matching.
- Stability is guaranteed only for the Householder form; more expressive RWKV-7-style matrices lose the norm bound.
Experimental setup
Datasets: FineWeb CodeParrot OpenThoughts-114k-Math TriviaQA lm-eval-harness tasks
Benchmarks: S3 S4 A5 S5 word problems FineWeb perplexity Lambada lm-eval-harness tasks CodeParrot OpenThoughts-114k-Math TriviaQA
Metrics: accuracy cross-entropy loss perplexity effective rank
Baselines: DeltaNet Gated DeltaNet RWKV-7 Mamba GLA
Builds on: DeltaNet Gated DeltaNet Mamba 2 RWKV-7
Closest papers by meaning
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues (ICLR 2025 · similarity 0.90)
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.89)
- Improving Bilinear RNN with Closed-loop Control (NeurIPS 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.87)
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.87)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.