Research map / Large language models / Retrieval and knowledge
Retrieval Augmented Language Model Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train REALM with latent retrieval over Wikipedia | REALM · NaturalQuestions-Open test | exact match: 33.3 -> 39.2 | Helped |
| Pre-train REALM with latent retrieval over Wikipedia | REALM · WebQuestions test | exact match: 36.4 -> 40.2 | Helped |
| Pre-train REALM with latent retrieval over Wikipedia | REALM · CuratedTrec test | exact match: 30.1 -> 46.8 | Helped |
| Use CC-News rather than Wikipedia as the unlabeled pre-training corpus | REALM · NaturalQuestions-Open test | exact match: 39.2 -> 40.4 | Helped |
| Use CC-News rather than Wikipedia as the unlabeled pre-training corpus | REALM · WebQuestions test | exact match: 40.2 -> 40.7 | Helped |
| Use CC-News rather than Wikipedia as the unlabeled pre-training corpus | REALM · CuratedTrec test | exact match: 46.8 -> 42.9 | Hurt |
| Add REALM pre-training to the encoder while keeping the baseline retriever (ablation) | REALM · NaturalQuestions-Open development | exact match: 31.3 -> 35.3 | Helped |
| Add REALM pre-training to the retriever while keeping the baseline encoder (ablation) | REALM · NaturalQuestions-Open development | exact match: 31.3 -> 37.4 | Helped |
| Use salient-span masking instead of random uniform token masking (ablation) | REALM · NaturalQuestions-Open development | exact match: 32.3 -> 38.2 | Helped |
| Use salient-span masking instead of random span masking (ablation) | REALM · NaturalQuestions-Open development | exact match: 35.3 -> 38.2 | Helped |
| Refresh the MIPS index frequently rather than using 30× stale MIPS (ablation) | REALM · NaturalQuestions-Open development | exact match: 28.7 -> 38.2 | Helped |
| Refresh the MIPS index frequently rather than using 30× stale MIPS (ablation) | REALM · NaturalQuestions-Open development | zero-shot retrieval Recall@5: 15.1 -> 38.5 | Helped |
Closest papers by meaning
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.90)
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.90)
- Reusing Pre-Training Data at Test Time is a Compute Multiplier (ICLR 2026 · similarity 0.88)
- Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-Hoc Retrieval (AAAI 2024 · similarity 0.88)
- A Statistical Framework for Data-dependent Retrieval-Augmented Models (ICML 2024 · similarity 0.88)
- RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.