Research map / Large language models / Retrieval and knowledge
InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructRAG, where an instruction-tuned LM self-synthesizes denoising rationales from retrieved documents and ground-truth answers, then uses them as in-context demonstrations (ICL) or SFT targets (FT) for RAG.
Problem
Retrieval-augmented generation is hurt by noisy or irrelevant retrieved documents, and existing RAG methods denoise only implicitly by predicting answers directly, while explicit denoising supervision is costly to obtain.
Approach
Use the LM's own instruction-following ability to generate rationales that identify useful documents and explain how they lead to the ground-truth answer, requiring no extra supervision; these rationales serve as ICL demonstrations or SFT targets so the model learns explicit denoising.
Key results
- InstructRAG-FT on Llama-3-Instruct-8B reaches 66.2 on PopQA, 78.5 on TriviaQA, 65.7 on NQ and 57.2 on 2WikiMultiHopQA, outperforming trainable baselines on most metrics.
- InstructRAG-ICL on Llama-3-Instruct-70B reaches 81.2 on TriviaQA and 66.5 on NQ, improving over training-free baselines.
- The paper reports an average relative improvement of 8.3% over the best baseline across five benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use self-synthesized denoising rationales with supervised fine-tuning rather than vanilla SFT | Llama-3-Instruct 8B RAG · PopQA | accuracy: 61.0 -> 66.2 | Helped |
| Use self-synthesized denoising rationales with supervised fine-tuning rather than vanilla SFT | Llama-3-Instruct 8B RAG · TriviaQA | accuracy: 73.9 -> 78.5 | Helped |
| Use self-synthesized denoising rationales with supervised fine-tuning rather than vanilla SFT | Llama-3-Instruct 8B RAG · Natural Questions | accuracy: 56.6 -> 65.7 | Helped |
| Use self-synthesized denoising rationales with supervised fine-tuning rather than vanilla SFT | Llama-3-Instruct 8B RAG · 2WikiMultiHopQA | accuracy: 56.1 -> 57.2 | Helped |
| Use self-synthesized denoising rationales with supervised fine-tuning rather than vanilla SFT | Llama-3-Instruct 8B RAG · ASQA | exact match: 43.8 -> 47.6 | Helped |
| Use a 70B rather than 8B rationale generator (ablation) | InstructRAG-FT · PopQA | accuracy: 66.2 -> 67.0 | Helped |
| Use a 70B rather than 8B rationale generator (ablation) | InstructRAG-FT · ASQA | exact match: 47.6 -> 49.1 | Helped |
| Replace LM-generated rationales with template-based rationales (ablation) | InstructRAG-FT · PopQA | accuracy: 66.2 -> 59.6 | Hurt |
| Replace LM-generated rationales with template-based rationales (ablation) | InstructRAG-FT · ASQA | exact match: 47.6 -> 46.3 | Hurt |
| Omit the ground-truth answer during rationale generation (ablation) | InstructRAG-FT · PopQA | accuracy: 66.2 -> 65.2 | Hurt |
| Omit retrieved documents during rationale generation (ablation) | InstructRAG-FT · ASQA | exact match: 47.6 -> 45.2 | Hurt |
| Add demonstrations at inference (ablation) | InstructRAG-FT · ASQA | exact match: 47.6 -> 44.7 | Hurt |
Surprising findings
- Self-RAG consistently underperforms vanilla SFT and even the training-free in-context RALM baseline across benchmarks.
- Demonstrations hurt InstructRAG-FT performance, while they help InstructRAG-ICL.
Limitations
- Evaluation focuses on question answering tasks; generalization to open-ended generation is unclear.
- Accuracy and exact-match metrics are pattern-based and biased toward longer generations.
- Overall rationale-answer consistency is only 89% on all training samples because some samples have no relevant retrieved documents.
Experimental setup
Datasets: PopQA TriviaQA Natural Questions ASQA 2WikiMultiHopQA HumanEval
Benchmarks: PopQA TriviaQA Natural Questions ASQA 2WikiMultiHopQA HumanEval
Metrics: accuracy exact match (str-em) citation precision citation recall pass@k
Baselines: In-Context RALM Few-shot demonstration with instruction Vanilla SFT RetRobust Self-RAG Llama-3-Instruct-8B zero-shot Llama-3-Instruct-70B zero-shot
Builds on: in-context RALM Self-RAG RetRobust chain-of-thought prompting
Closest papers by meaning
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.89)
- Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection (ICLR 2025 · similarity 0.89)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.88)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.88)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.88)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.