Research map / Architectures and efficiency / Attention and transformers

Titans: Learning to Memorize at Test Time

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 18 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Titans, a family of architectures built on a deep neural long-term memory that learns to memorize at test time via a surprise-driven, momentum-and-decay associative memory update, with three ways to combine it with attention (MAC, MAG, MAL).

Problem

Transformers have quadratic cost and limited context, while linear recurrent models compress long history into small vector or matrix states that cannot hold very long contexts; existing memory designs lack surprise-based updates, forgetting, and deep memory.

Approach

A deep MLP long-term memory is trained in the inner loop at test time to store key-value associations, with surprise defined as the gradient of an associative memory loss, combined with past-surprise momentum and adaptive weight decay; attention acts as short-term memory and the neural memory as long-term memory, combined via MAC, MAG, or MAL variants. Training is parallelized by chunking into mat

Key results

• At 760M params / 30B tokens, Titans (MAC) reaches 52.51 average commonsense reasoning accuracy vs 48.69 for Transformer++ and 49.69 for Gated DeltaNet.
• On S-NIAH-PK at 16K, Titans (LMM) scores 96.2 vs 88.4 for TTT and 71.4 for DeltaNet.
• Titans (LMM) achieves best time-series MSE on ETTm1 (0.358), ETTm2 (0.261), ETTh1 (0.420), ETTh2 (0.336), ECL (0.162), and Weather (0.231) among compared models.

Measured effects reported in the paper

Surprising findings

• Hybrid MAG and MAC variants outperform the MAL-style hybrid, which is the layout most hybrid models in the literature use.

Limitations

• Memory is a simple MLP; more expressive memory architectures are left for future work
• Larger model results were deferred to a later version of the paper
• Deeper memory increases training cost, showing an effectiveness-efficiency trade-off

Experimental setup

Datasets: FineWeb-Edu Pile subset Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c SIQA BoolQ RULER S-NIAH

Benchmarks: Wikitext LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c SIQA BoolQ RULER S-NIAH BABILong GenomicsBenchmarks

Metrics: perplexity accuracy MSE MAE training throughput

Baselines: Transformer++ RetNet GLA Mamba Mamba2 DeltaNet TTT Gated DeltaNet Samba

Builds on: linear transformers DeltaNet Gated DeltaNet TTT Mamba2 Recurrent Memory Transformer

Closest papers by meaning

• Proteus: Incremental Memory Activation for Long-Context Sequence Modeling (NeurIPS 2026 · similarity 0.88)
• Learning What to Remember: Test-Time Training via Context Distillation (NeurIPS 2026 · similarity 0.88)
• Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.88)
• Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.87)
• Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.87)
• Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.86)