Research map / Architectures and efficiency / Attention and transformers

RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RAM-Net, a fixed-size slot-array recurrent state accessed via sparse product-softmax address decoding with Top-K selection, gated sparse updates and cyclic address positional embeddings, achieving strong long-context retrieval and perplexity at 340M/1.3B scales with far fewer state elements accessed per step.

Problem

Linear attention compresses history into one shared recurrent state, so information from different tokens superposes, causing inter-token interference that degrades fine-grained long-range recall; enlarging the state also raises per-step access cost.

Approach

Organize the recurrent state as a fixed array of independent slots and let each key/query select a sparse Top-K set of slots via a Product Softmax address decoder (computed by exact prefix-tree search), with a gated sparse update that writes and forgets only addressed slots, plus cyclic address positional embedding.

Key results

• At 340M, RAM-Net Top-32 reaches the lowest WikiText perplexity (25.87 vs 26.23 for Mamba2) and S-NIAH average 72.9 vs 51.3 for Gated DeltaNet, while Top-8 accesses 0.4M state elements per step vs 12.9M for Mamba2.
• At 1.3B, RAM-Net Top-16 achieves WikiText perplexity 15.90, edging Mamba2 (15.96), with S-NIAH average 70.1 vs 62.4 for Mamba2.
• RAM-Net Top-8 at 340M achieves 64.7 average S-NIAH vs 35.0 for Mamba2, with 8x fewer active state elements per step than Mamba2.

Surprising findings

• Removing output RMSNorm collapses S-NIAH-2 accuracy from 83.7 to 16.3 despite slots already being bounded averages.
• An MoE-style load-balancing loss on write addresses hurts retrieval (S-NIAH-2/3 drop to 11.3/5.7), suggesting uneven slot usage is beneficial.

Limitations

• Experiments limited to models up to 1.3B parameters.
• Sparse addressing trades off recall when query and relevant content are not co-addressed, especially with diverse distractors over long ranges.
• GSU updates each addressed slot independently; coordinated joint updates are not explored.

Experimental setup

Datasets: FineWeb-Edu RULER S-NIAH MQAR (Zoology) WikiText-103 LAMBADA MMLU ARC OpenBookQA SciQ COPA PIQA HellaSwag

Benchmarks: WikiText-103 LAMBADA MMLU ARC-e ARC-c OpenBookQA SciQ boolQ COPA PIQA HellaSwag WinoGrande

Metrics: perplexity accuracy decoding throughput per-step active state size

Baselines: Transformer++ Mamba2 GLA GSA Gated DeltaNet DeltaNet Linear attention SWA H3

Builds on: Gated DeltaNet Mamba2 GLA GSA RoPE Flame

Closest papers by meaning

• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.91)
• Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.91)
• Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access (NeurIPS 2025 · similarity 0.89)
• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.89)
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.88)
• Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR 2025 · similarity 0.87)