Research map / Architectures and efficiency / Attention and transformers
Memorizing Transformers
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, Christian Szegedy
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 65K external memory | Transformer-XL · arXiv Math | perplexity: 2.42 -> 2.26 | Improved |
| Add 65K external memory | Transformer-XL · PG-19 | perplexity: 11.88 -> 11.37 | Improved |
| Add 65K external memory | Transformer-XL · C4(4K+) | perplexity: 14.03 -> 13.64 | Improved |
| Add 65K external memory | Transformer-XL · GitHub | perplexity: 2.10 -> 1.80 | Improved |
| Add 65K external memory | Transformer-XL · Isabelle | perplexity: 2.16 -> 1.99 | Improved |
| Increase memory size from 8192 to 65K | Memorizing Transformer, context 512 with XL cache 512 · arXiv Math | perplexity: 2.37 -> 2.31 | Improved |
| Increase memory size from 8192 to 65K | Memorizing Transformer, context 512 with XL cache 512 · PG-19 | perplexity: 11.93 -> 11.62 | Improved |
| Increase memory size from 8192 to 65K | Memorizing Transformer, context 512 with XL cache 512 · GitHub | perplexity: 2.03 -> 1.87 | Improved |
| Increase memory size from 8192 to 65K | Memorizing Transformer, context 512 with XL cache 512 · C4(4K+) | perplexity: 14.04 -> 14.04 | No clear effect |
| Use two kNN retrieval layers rather than one (ablation) | Memorizing Transformer · Language modeling tasks | perplexity | No clear effect |
| Place the memory layer at layer 3 rather than layer 9 (ablation) | Memorizing Transformer · arXiv Math | perplexity: 2.37 -> 2.40 | Hurt |
| Place the memory layer at layer 6 rather than layer 9 (ablation) | Memorizing Transformer · arXiv Math | perplexity: 2.37 -> 2.36 | Improved |
Closest papers by meaning
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.89)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.88)
- Analyzing the Power of Chain of Thought through Memorization Capabilities (NeurIPS 2025 · similarity 0.87)
- Mitigating Memorization in Language Models (ICLR 2025 · similarity 0.87)
- MEMORYLLM: Towards Self-Updatable Large Language Models (ICML 2024 · similarity 0.87)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
