Research map / Architectures and efficiency / Attention and transformers

Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes incremental memory activation (Proteus), a position-scheduled block-wise gating that progressively unlocks fixed-size recurrent memory (or MLP parameters) over the context, applied to SWLA, Comba, Titans and Hope-Attention.

Problem

Static memory in recurrent/memory-based sequence models gets polluted by early tokens and saturates, hurting retention of later context in long-context modeling.

Approach

Expose only a growing prefix of fixed-size memory blocks as the context grows (early bottleneck, then fresh capacity unlocked), gating both reads and writes at no added parameters; extended to MLP parameters via Nested Learning for Hope-Attention.

Key results

• Titans+Proteus average commonsense accuracy at 1.3B rises from 56.95 to 58.00, with Wikitext perplexity 15.36 to 14.94.
• Titans+Proteus on 16K S-NIAH-3 improves accuracy from 21.4 to 29.8.
• Hope-Attention+Proteus at 760M improves average commonsense accuracy from 53.15 to 53.99 and LAMBADA perplexity from 21.36 to 19.72.

Surprising findings

• At 1.3B, Hope-Attention+Proteus has slightly higher Wikitext perplexity (15.70 vs 15.66) than the base model.
• Gains on LongBench are small for Comba (13.05 to 13.23) and some tasks decline, e.g., Hotpot for Comba.

Limitations

• Activation schedule is fixed and hand-designed (uniform growth with position), not optimized or data-dependent.
• Extension to MLP blocks (Hope-Attention) is a proof of concept on a single architecture.
• Gains concentrate at longer contexts and harder tasks; neutral where base models already saturate.

Experimental setup

Datasets: FineWeb

Benchmarks: Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c SIQA BoolQ SWDE SQuAD FDA

Metrics: perplexity zero-shot accuracy accuracy vs context length

Baselines: Transformer++ RetNet SWLA DeltaNet Comba Titans Hope-Attention

Builds on: Titans Nested Learning DeltaNet Hope

Closest papers by meaning

• Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.89)
• Titans: Learning to Memorize at Test Time (NeurIPS 2025 · similarity 0.88)
• Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.86)
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.86)
• Predictive Attractor Models (NeurIPS 2024 · similarity 0.86)
• On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models (NeurIPS 2026 · similarity 0.86)