Research map / Learning theory and algorithms / Prediction and decision losses
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Neehal Tumma, Noel Loo, Daniela Rus
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diagonal preconditioner (PGDN) to Gated DeltaNet | GDN 340M · commonsense reasoning average | accuracy: 48.86 -> 49.96 | Improved |
| add diagonal preconditioner (PKDA) to Kimi Delta Attention | KDA 340M · commonsense reasoning average | accuracy: 49.12 -> 49.89 | Improved |
| add diagonal preconditioner (PDN) to DeltaNet | DeltaNet 340M · commonsense reasoning average | accuracy: 47.41 -> 47.96 | Improved |
| add diagonal preconditioner to DeltaNet | DeltaNet 340M · in-context retrieval average | accuracy: 22.59 -> 23.84 | Improved |
| add diagonal preconditioner to GDN | GDN 340M · in-context retrieval average | accuracy: 24.77 -> 26.17 | Improved |
| add diagonal preconditioner to KDA | KDA 340M · in-context retrieval average | accuracy: 25.88 -> 26.76 | Improved |
| add diagonal preconditioner to GDN at 1B scale | GDN 1B · commonsense reasoning and in-context retrieval averages | accuracy: 55.44 -> 56.38 (CR); 33.87 -> 34.99 (ICR) | Improved |
| add diagonal preconditioner to KDA at 1B scale | KDA 1B · commonsense reasoning and in-context retrieval averages | accuracy: 55.43 -> 56.64 (CR); 34.55 -> 34.88 (ICR) | Improved |
| add preconditioner to DeltaNet and GDN | DN, GDN 340M · MQAR synthetic recall | accuracy | Improved |
| add preconditioner to DeltaNet, GDN, KDA | 340M models · S-NIAH-1/2/3 (RULER) | accuracy | Improved |
| apply preconditioner to the write key (ATK on GDN) instead of the query (ATQ on Mamba-2) (ablation) | preconditioned recurrence · training loss, 340M | train loss: 2.543 -> 2.536 | Improved |
| use stable squashed preconditioner parameterization Bt instead of raw inverse of At (ablation) | PGDN · training loss, 340M | train loss: 2.536 -> 2.511 | Improved |
Closest papers by meaning
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.88)
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.87)
- Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.87)
- Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025 · similarity 0.86)
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.86)
- Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
