Research map / Architectures and efficiency / Attention and transformers
Improving Language Models by Retrieving from Trillions of Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale the retrieval database from Wikipedia to full MassiveText | RETRO · Wikitext103 valid | perplexity: 18.46 -> 3.21 | Helped |
| add retrieval from Wikipedia to the baseline transformer | Baseline transformer · Wikitext103 valid | perplexity: 21.53 -> 18.46 | Helped |
| fine-tune with DPR retrieval passages | RETRO 7.5B · Natural Questions test | exact match accuracy: 30.4 -> 45.5 | Helped |
| remove query conditioning from the retrieval encoder (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.829 | Hurt |
| remove cross-attention positional encodings (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.826 | Hurt |
| share encoder and decoder embeddings (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.823 | No clear effect |
| use a 6-layer retrieval encoder (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.821 | No clear effect |
| attend to neighbour chunks only, omitting continuations (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.950 | Hurt |
| attend to continuation chunks only, omitting neighbours (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.895 | Hurt |
| remove retrieval (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.987 | Hurt |
| train with 1 neighbour instead of 2 (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.858 | Hurt |
| train with 4 neighbours instead of 2 (ablation) | RETRO · C4 eval | bpb: 0.822 -> 0.847 | No clear effect |
Closest papers by meaning
- BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models (ICLR 2024 · similarity 0.88)
- Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.88)
- One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.88)
- InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining (ICML 2024 · similarity 0.87)
- Language Model Memory and Memory Models for Language (NeurIPS 2026 · similarity 0.87)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.