Research map / Large language models / Retrieval and knowledge
RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RA-DIT, a lightweight two-stage dual instruction tuning method that retrofits LLaMA with retrieval by fine-tuning the LM on retrieval-augmented instructions and the Dragon+ query encoder with LM-supervised retrieval (KL to LM preferences).
Problem
Retrieval-augmented LMs are costly to build via retrieval-specific pre-training, while off-the-shelf fusion of LLMs and retrievers is suboptimal because LLMs are not trained to use retrieved content.
Approach
Decouple retrieval augmentation into two lightweight fine-tuning steps: retrieval-augmented instruction tuning of the LM (RA-IT, with parallel in-context retrieval and label/next-token loss) and LM-supervised retriever fine-tuning that updates only the query encoder to match the LM's chunk preferences via KL divergence, using both corpus text and multi-task instruction data.
Key results
- RA-DIT 65B reaches 49.1 average 0-shot on knowledge-intensive tasks versus 43.1 for LLaMA 65B + RePlug, and 55.2 vs 54.4-ish 5-shot gap of +1.4 points over RePlug on the four-task average.
- In the 64-shot setting, RA-DIT 65B averages 60.9 vs 56.8 for Atlas 11B across 8 KILT tasks.
- Retriever fine-tuning with 95% corpus and 5% MTI data improves the 6-task KILT dev average from 56.6 to 57.0 over base Dragon+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply dual instruction tuning to Llama 65B with Dragon+ | Llama 65B + Dragon+ · Knowledge-intensive tasks, 0-shot (MMLU, NQ, TQA, ELI5) | Avg: 45.1 -> 49.1 | Helped |
| Apply dual instruction tuning to Llama 65B with Dragon+ | Llama 65B + Dragon+ · Knowledge-intensive tasks, 5-shot (MMLU, NQ, TQA, ELI5) | Avg: 51.1 -> 51.8 | Helped |
| Apply retrieval-augmented instruction tuning | Llama 65B · Commonsense reasoning dev tasks, 0-shot, without retrieval | Avg: 72.1 -> 74.5 | Helped |
| Add LM fine-tuning and retriever fine-tuning together (ablation) | Llama 65B + Dragon+ · KILT dev tasks, 5-shot | Avg: 53.8 -> 54.6 | Helped |
| Add retrieval-augmented LM fine-tuning without retriever fine-tuning (ablation) | Llama 65B + Dragon+ · KILT dev tasks, 5-shot | Avg: 53.8 -> 53.9 | Helped |
| Fine-tune the retriever without LM fine-tuning (ablation) | Llama 65B + Dragon+ · KILT dev tasks, 5-shot | Avg: 53.8 -> 54.4 | Helped |
| Fine-tune the retriever with 95% corpus data and 5% MTL data (ablation) | Llama 65B + Dragon+ · KILT dev tasks, 5-shot | Avg: 56.6 -> 57.0 | Helped |
| Fine-tune the retriever with MTL instruction data only (ablation) | Llama 65B + Dragon+ · KILT dev tasks, 5-shot | Avg: 56.6 -> 56.4 | No clear effect |
| Fine-tune both query and document encoders instead of only the query encoder (ablation) | Dragon+ retriever · KILT dev tasks, 5-shot | Avg: 57.8 -> 55.8 | Hurt |
| Concatenate retrieved chunks instead of using parallel retrieval augmentation (ablation) | RA-IT 65B · KILT dev tasks, 0-shot, top-3 chunks | Avg: 50.1 -> 50.9 | Helped |
| Increase retrieved chunks from top-1 to top-10 (ablation) | Llama 65B + Dragon+ · KILT dev tasks, 5-shot | Avg: 51.2 -> 53.4 | Helped |
| Use Dragon+ instead of Contriever (ablation) | Llama 65B with retrieval augmentation · KILT dev tasks, 5-shot | Avg: 47.5 -> 53.4 | Helped |
Surprising findings
- Fine-tuning both encoders of the retriever hurts performance relative to fine-tuning only the query encoder.
- Conventional instruction tuning gives marginal or negative gains in the 5-shot setting for most knowledge tasks.
Limitations
- Iterative dual instruction tuning did not yield further gains in preliminary experiments.
- Retrieval queries are manually constructed templates rather than automatically generated.
- Parallel retrieval augmentation is computationally expensive when prompts share long prefixes, as in few-shot settings.
Experimental setup
Datasets: OpenAssistant Conversations (oasst1) CommonsenseQA MathQA Web Questions Wiki QA SQuAD v2 MS MARCO CoQA DROP CNN/DailyMail GSM8K MATH
Benchmarks: MMLU Natural Questions TriviaQA ELI5 HotpotQA FEVER AIDA CoNLL-YAGO Zero-Shot RE T-REx Wizard of Wikipedia BoolQ PIQA
Metrics: accuracy exact match Rouge-L F1
Baselines: LLaMA 65B RePlug (LLaMA 65B + Dragon+) Atlas 11B Contriever Contriever-msmarco IT 65B (conventional instruction tuning)
Builds on: RePlug In-Context RALM LSR (LM-supervised retrieval) Dragon+ Atlas
Closest papers by meaning
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.89)
- InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining (ICML 2024 · similarity 0.89)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.88)
- Improving Retrieval Augmented Language Model with Self-Reasoning (AAAI 2025 · similarity 0.88)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.87)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.