Research map / Architectures and efficiency / Attention and transformers

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Sparse Delta Memory (SDM), a sparsified Gated DeltaNet whose large state is addressed via product-key top-k slot selection, giving much larger state at iso-FLOPs and better long-context recall than GDN.

Problem

Linear RNNs/linear attention have small fixed states that limit long-context recall, while increasing state size in dense models like Gated DeltaNet raises FLOPs linearly.

Approach

Sparsify the Gated DeltaNet update by maintaining an N-slot memory table and applying gated delta writes and reads only to top-W write and top-R read slots chosen via product-key indexing, so state size grows without increasing per-token FLOPs; the memory is also learned as an initial state M0 for storing pretraining knowledge.

Key results

• At 8B, SDM reaches average RULER accuracy 50.2 vs 34.2 for GDN (FullAttn 61.2); at 1.4B, 31.2 vs 20.0 for GDN.
• At 8B, SDM achieves validation text NLL 2.253 vs 2.298 for GDN and 2.285 for FullAttn, under iso-FLOP matching.
• SDM state at 8B is roughly 4,000x larger than GDN's state (7.963B vs 2.2M state values), with training throughput 1.49x slower than GDN.

Surprising findings

• Learning the initial state M0 helps SDM substantially (RULER 28.0 to 31.2 at 1.4B) but does not help GDN, which has a tiny state.
• SDM outperforms full attention on validation NLL and average short-context accuracy at 8B despite a fixed-size state.

Limitations

• SDM kernel MFU is around an order of magnitude lower than the optimized GDN FLA kernel; state resides in HBM.
• SDM memory footprint may be as large as model parameters, unsuitable for resource-constrained settings.
• Multikey_2 RULER task remains challenging; FullAttn retains a large advantage there.

Experimental setup

Datasets: RULER DCLM HellaSWAG WinoGrande ARC PIQA OpenBookQA RACE CommonsenseQA BoolQ TriviaQA HumanEval+

Benchmarks: RULER HellaSWAG WinoGrande ARC easy ARC challenge PIQA OpenBookQA RACE CommonsenseQA BoolQ TriviaQA HumanEval+

Metrics: validation NLL exact-match accuracy accuracy perplexity training loss

Baselines: Gated DeltaNet Mamba2 FullAttn (softmax attention) Fast-Weight Product Key Memory

Builds on: Gated DeltaNet DeltaNet Product Key Memory Mamba2 Sparse Access Memory

Closest papers by meaning

• RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.91)
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.88)
• Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.88)
• Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025 · similarity 0.88)
• Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.88)
• Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access (NeurIPS 2025 · similarity 0.87)