Research map / Architectures and efficiency / Attention and transformers

Raven: High-Recall Sequence Modeling via Sparse Memory Routing

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Raven, a linear-time sequence model with a fixed set of memory slots where a sparse, input-dependent top-K router selects which slots to decay and write, interpolating between SSMs and sliding-window attention.

Problem

Linear-time sequence models struggle with long-context recall: dense state updates in SSMs cause interference, while sliding-window attention loses tokens once they leave the window.

Approach

Routing Slot Memories: a slot-separable recurrence where a sparse top-K sigmoid router (with Gumbel noise in training) selects which memory slots are decayed and overwritten per token, combined with Mamba-2-style scalar decay applied only to selected slots, and no short convolutions.

Key results

• At 400M scale, Raven keeps over 91% accuracy on NIAH-1 at 32K (16x its 2K training length), the only model at that scale above 91%.
• On SWDE, Raven reaches 34.1% versus 25.7% for Mamba-2 and 29.0% for GLA at 400M.
• At 800M, Raven retains 91.0% on NIAH-1 at 64K where GDN drops to 45.2%.

Surprising findings

• Raven attains strong retrieval without short convolutions, which prior linear models rely on.
• Omitting the load-balancing loss is intentional; uneven slot usage emerges and yields dedicated retrieval slots.

Limitations

• Language modeling average is not clearly better: at 800M Raven trails GDN (43.3 vs 45.1 avg) and at 400M it is only on par with baselines.
• Evaluation is limited to small models (400M and 800M) trained on 15B/32B tokens.
• Router design choices such as Gumbel noise and the normalization constant alpha are hand-tuned per scale.

Experimental setup

Datasets: SlimPajama-627B SQuAD SWDE FDA RULER S-NIAH-1/2/3 PIQA Winogrande HellaSwag ARC-e ARC-c LAMBADA

Benchmarks: SWDE FDA SQuAD NIAH-1 NIAH-2 NIAH-3 LAMBADA PIQA HellaSwag Winogrande ARC-e ARC-c

Metrics: accuracy perplexity

Baselines: GLA GSA GDN Mamba-2 SWA FoX Transformer RoPE ABC SWA-RoPE

Builds on: Mamba-2 Gated Slot Attention Attention with Bounded Memory Control Sliding Window Attention DeepSeekMoE

Closest papers by meaning

• RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.91)
• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.89)
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.88)
• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.88)
• Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.88)
• Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention (NeurIPS 2025 · similarity 0.87)