Research map / Architectures and efficiency / Attention and transformers
Recurrent Memory Transformer
Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase BPTT unroll from 0 to 3 (ablation) | RMT mem 10 · WikiText-103 seg 150 | perplexity: 26.85 -> 25.04 | Improved |
| increase BPTT unroll from 0 to 1 for mem 25 (ablation) | RMT mem 25 · WikiText-103 seg 150 | perplexity: 29.73 -> 24.91 | Improved |
| increase BPTT unroll from 0 to 4 (ablation) | RMT mem 10 · WikiText-103 seg 50 | perplexity: 30.69 -> 26.25 | Improved |
| increase BPTT unroll from 0 to 2 (ablation) | RMT mem 20 · enwik8 | bpc: 1.301 -> 1.222 | Improved |
| add RMT memory to Tr-XL cache | Tr-XL · WikiText-103 seg 150 | perplexity: 24.12 -> 23.99 | Improved |
| replace Tr-XL cache with RMT memory (mem 10 vs 75) | Tr-XL · WikiText-103 seg 150 | perplexity: 24.68 (Tr-XL 75) vs 25.04 (RMT 10) | No clear effect |
| replace Memory Transformer with RMT recurrence | MemTr · WikiText-103 seg 150 | perplexity: 29.63 -> 25.04 | Improved |
| use RMT instead of Tr-XL on multi-segment quadratic equations | Tr-XL · Quadratic equations | accuracy: 0.93 -> 0.99 | Improved |
| increase number of segments in Copy/Reverse | Transformer-XL vs RMT · Copy/Reverse | accuracy | Improved |
| increase memory size from 5 to 50 (ablation) | RMT · WikiText-103 seg 50 | perplexity | No clear effect |
| add RMT memory of 10 tokens to pretrained models with up to 4 segments | RoBERTa-base · Hyperpartisan news | F1: 94.87 -> 98.11 | Improved |
| add RMT memory with more segments | BERT / DeBERTa · Hyperpartisan news | F1: deberta 96.78 (2 seg) -> 94.80 (3-4 seg) | Mixed |
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.92)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.89)
- Cached Transformers: Improving Transformers with Differentiable Memory Cachde (AAAI 2024 · similarity 0.88)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.88)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.88)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
