Research map / Architectures and efficiency / Attention and transformers
Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments pre-trained transformers (BERT, GPT-2, GPT-Neo) with token-based recurrent memory and segment-level curriculum learning to process inputs up to 2M tokens with linear compute scaling.
Problem
Quadratic scaling of self-attention limits transformer input length, and most recurrent long-context methods require architectural changes or training from scratch.
Approach
Wrap any pretrained transformer with a set of trainable memory vectors prepended to each segment and passed recurrently to the next segment, with no architectural changes, trained via BPTT under a segment-wise curriculum.
Key results
- RMT with BERT stores task facts across 7 training segments and retrieves them at up to 4096 segments (2,048,000 tokens) with high accuracy.
- On ArXiv language modeling, RMT with GPT-2 (2 memory tokens) gives lower perplexity than the baseline GPT-2 trained for the same number of steps, and perplexity improves as more history segments are processed.
- RMT reduces inference FLOPs by up to x295 relative to OPT-135M-style full attention at 2,048,000 tokens (OPT-175B: x29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add recurrent memory and train with increasing input context | GPT-2 · ArXiv language modeling | perplexity | Helped |
| Add recurrent memory | GPT-Neo · Formal theorem proving | perplexity | Helped |
| Increase the number of training segments to 5 or more (ablation) | RMT with BERT · Synthetic memorization, detection, and reasoning tasks | generalization to longer sequences | Helped |
| Train with history across multiple segments instead of without history (ablation) | RMT with GPT-2 · ArXiv language modeling | perplexity on longer contexts | Helped |
| Mix the number of segments at each curriculum step (ablation) | RMT with GPT-2 · ArXiv language modeling | generalization to other sequence lengths | Helped |
| Train on 4 or more segments (ablation) | RMT with GPT-Neo · Formal theorem proving | predictions for longer sequences | No clear effect |
Surprising findings
- Models trained on 5 or more segments generalize nearly perfectly to tasks twice as long, and hold up on up to 4096 segments despite training on at most 7.
- Memory operations emerge without explicit memory-read/write supervision; attention patterns reveal write, retain, and read behaviors.
Limitations
- Scaling to 2M tokens demonstrated only on synthetic tasks; no established long NLP benchmarks at that length.
- BPTT training remains computationally expensive and curriculum choice strongly affects generalization.
- Compressed memory states may underperform full-attention models on tasks needing exact past representations.
Experimental setup
Datasets: bAbI QuALITY The Pile (ArXiv) Lean 3 Mathlib
Benchmarks: bAbI-based memorization tasks (Memorize, Detect & Memorize, Reasoning) ArXiv language modeling perplexity Lean 3 lemma-based proof perplexity
Metrics: accuracy perplexity FLOPs
Baselines: GPT-2 without memory GPT-Neo without memory OPT Transformer-XL Memformer
Builds on: Recurrent Memory Transformer Memory Transformer Transformer-XL
Closest papers by meaning
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.93)
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.90)
- Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract) (AAAI 2024 · similarity 0.90)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.90)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.90)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.