AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Pretraining with hierarchical memories: separating long-tail and common knowledge

Hadi Pouransari, David Grangier, C Thomas, Michael Kirchhof, Oncel Tuzel

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
co-train anchor with memory instead of freezing anchor (ablation)160M anchor + memories · Avg-SKaccuracy (%): 39.2 -> 40.3Improved
train anchor and memories together from scratch instead of from pretrained anchor (ablation)160M anchor + memories · Avg-SKaccuracy (%): 40.3 -> 39.6Hurt
retrieve context-based memories instead of a generic same-size memory (ablation)160M anchor + memories · Avg-SKaccuracy (%): 35.7 -> 40.3Improved
add fetched memories (~10% params) to anchor1.4B anchor · Avg-SKaccuracy (%): 49.7 -> 54.9Improved
increase bank size via level-4 memory at same fetch size (ablation)hierarchical memories on 160M anchor · Avg-SKaccuracy (%): 39.1 -> 40.1Improved
use FFN memories instead of LoRA or KV memories (ablation)160M anchor memories · Specific-knowledge benchmarksaccuracyImproved
replace MiniLM with GTE-small embeddings for clustering (ablation)160M anchor + memories · Avg-SKaccuracy (%): 36.6 -> 36.2No clear effect
use 2-level clustering with k=256 instead of 4-level k=16 (ablation)160M anchor + memories · Avg-SKaccuracy (%): 36.6 -> 37.5Improved
use vanilla RAG over DCLM instead of memories160M-1.4B baselines · Avg-SK 0-shotaccuracy (%): 46.9 -> 46.1 (1.4B)Hurt
add post-hoc memories to Qwen 2.5 0.5BQwen 2.5 0.5B · Avg-CKaccuracy (%): Avg-SK 40.6 -> 44.5; Avg-CK 53.9 -> 52.1Mixed
block 1/16 of memory bank from retrieval (ablation)410M anchor + memories · atomic number accuracyaccuracy (%): 70% -> 20%Hurt
add memories on pretrained open-weight models post-hocGemma 3 270M · Atomic number accuracyaccuracy (%): 1.7 -> 49.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy