Research map / Architectures and efficiency / Attention and transformers
Pretraining with hierarchical memories: separating long-tail and common knowledge
Hadi Pouransari, David Grangier, C Thomas, Michael Kirchhof, Oncel Tuzel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| co-train anchor with memory instead of freezing anchor (ablation) | 160M anchor + memories · Avg-SK | accuracy (%): 39.2 -> 40.3 | Improved |
| train anchor and memories together from scratch instead of from pretrained anchor (ablation) | 160M anchor + memories · Avg-SK | accuracy (%): 40.3 -> 39.6 | Hurt |
| retrieve context-based memories instead of a generic same-size memory (ablation) | 160M anchor + memories · Avg-SK | accuracy (%): 35.7 -> 40.3 | Improved |
| add fetched memories (~10% params) to anchor | 1.4B anchor · Avg-SK | accuracy (%): 49.7 -> 54.9 | Improved |
| increase bank size via level-4 memory at same fetch size (ablation) | hierarchical memories on 160M anchor · Avg-SK | accuracy (%): 39.1 -> 40.1 | Improved |
| use FFN memories instead of LoRA or KV memories (ablation) | 160M anchor memories · Specific-knowledge benchmarks | accuracy | Improved |
| replace MiniLM with GTE-small embeddings for clustering (ablation) | 160M anchor + memories · Avg-SK | accuracy (%): 36.6 -> 36.2 | No clear effect |
| use 2-level clustering with k=256 instead of 4-level k=16 (ablation) | 160M anchor + memories · Avg-SK | accuracy (%): 36.6 -> 37.5 | Improved |
| use vanilla RAG over DCLM instead of memories | 160M-1.4B baselines · Avg-SK 0-shot | accuracy (%): 46.9 -> 46.1 (1.4B) | Hurt |
| add post-hoc memories to Qwen 2.5 0.5B | Qwen 2.5 0.5B · Avg-CK | accuracy (%): Avg-SK 40.6 -> 44.5; Avg-CK 53.9 -> 52.1 | Mixed |
| block 1/16 of memory bank from retrieval (ablation) | 410M anchor + memories · atomic number accuracy | accuracy (%): 70% -> 20% | Hurt |
| add memories on pretrained open-weight models post-hoc | Gemma 3 270M · Atomic number accuracy | accuracy (%): 1.7 -> 49.2 | Improved |
Closest papers by meaning
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.90)
- Memory Layers at Scale (ICML 2025 · similarity 0.90)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.89)
- Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract) (AAAI 2024 · similarity 0.88)
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.88)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
