Research map / Label-efficient and robust learning / Knowledge distillation
Distilling Knowledge from Reader to Retriever for Question Answering
Gautier Izacard, Edouard Grave
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill reader cross-attention scores into a retriever, initialized from DPR | Fusion-in-Decoder with T5 large · NaturalQuestions test | EM: 51.4 -> 54.4 | Improved |
| Distill reader cross-attention scores into a retriever, initialized from DPR | Fusion-in-Decoder with T5 large · TriviaQA test | EM: 67.6 -> 72.5 | Improved |
| Initialize the retriever training passages from DPR rather than BM25 | Proposed reader-to-retriever distillation · NaturalQuestions test | R@20: 80.0 -> 84.3 | Improved |
| Initialize the retriever training passages from DPR rather than BM25 | Proposed reader-to-retriever distillation · TriviaQA test | R@20: 81.4 -> 83.6 | Improved |
| Perform an additional self-training iteration | Proposed method initialized with BM25 · NaturalQuestions validation | Dev EM: 41.2 -> 46.8 -> 47.9 -> 46.2 | Mixed |
| Perform an additional self-training iteration | Proposed method initialized with BM25 · TriviaQA validation | Dev EM: 65.3 -> 66.7 -> 67.5 -> 67.7 | Improved |
| Use KL-divergence rather than mean squared error to train the retriever (ablation) | Proposed method initialized with BM25 · NaturalQuestions validation | R@20: 61.2 -> 76.4 | Improved |
| Use KL-divergence rather than max-margin loss, gamma=1 (ablation) | Proposed method initialized with BM25 · NaturalQuestions validation | R@20: 73.6 -> 76.4 | Improved |
| Use KL-divergence rather than max-margin loss, gamma=0.2 (ablation) | Proposed method initialized with BM25 · NaturalQuestions validation | R@20: 73.5 -> 76.4 | Improved |
| Use KL-divergence rather than max-margin loss, gamma=0.1 (ablation) | Proposed method initialized with BM25 · NaturalQuestions validation | R@20: 73.5 -> 76.4 | Improved |
| Average attention over the first output token only rather than all output tokens (ablation) | Proposed method initialized with BM25 · NaturalQuestions validation | R@20: 75.3 -> 76.4 | Improved |
| Average attention over passage tokens rather than taking their maximum (ablation) | Proposed method initialized with BM25 · NaturalQuestions validation | R@20: 72.5 -> 76.4 | Improved |
Closest papers by meaning
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.86)
- The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation (NeurIPS 2025 · similarity 0.86)
- A Statistical Framework for Data-dependent Retrieval-Augmented Models (ICML 2024 · similarity 0.85)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.85)
- Fine-Grained Distillation for Long Document Retrieval (AAAI 2024 · similarity 0.85)
- UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
