Research map / Large language models / Retrieval and knowledge
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Speculative RAG, where a small instruction-tuned Mistral-7B drafter generates multiple answer-plus-rationale drafts in parallel from document subsets, and a larger untuned generalist LM (Mistral-7B or Mixtral-8x7B) verifies them via language-modeling scores.
Problem
Standard RAG with many retrieved documents incurs long input contexts, high latency, and position bias, while existing improvements (Self-RAG, CRAG) require instruction-tuning generalist LMs or add latency.
Approach
Decompose RAG into drafting by a small specialist LM over diverse, clustered document subsets (generating rationale plus answer in parallel), followed by a single verification pass by an off-the-shelf generalist LM that scores drafts using self-consistency, self-reflection, and draft probability, adapting speculative decoding to answer level.
Key results
- M_Verifier-8x7B + M_Drafter-7B reaches 74.24 on TriviaQA, 31.57 on MuSiQue, 57.54 on PopQA, 76.60 on PubHealth, and 80.55 on ARC-Challenge, beating the best standard RAG baseline Mixtral-Instruct-8x7B.
- Improves PubHealth accuracy by up to 12.97% over Mixtral-Instruct-8x7B while reducing latency by 50.83% versus the standard RAG baseline.
- Random selection without verification drops accuracy by 5.69% on TriviaQA, showing the verifier's contribution.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pair the instruction-tuned drafter with the Verifier-8x7B | Speculative RAG · TriviaQA, MuSiQue, PopQA, PubHealth, ARC-Challenge | accuracy: 0.33%; 2.15%; 3.86%; 12.97%; 2.14% | Helped |
| Replace multi-perspective sampling with random document sampling (ablation) | Speculative RAG drafting · TriviaQA and PubHealth | accuracy: -1.22; -1.22 | Hurt |
| Sample all documents from one perspective cluster instead of diverse clusters (ablation) | Speculative RAG drafting · TriviaQA and PubHealth | accuracy: -1.88; -2.23 | Hurt |
| Add rho_Draft to verification scoring (ablation) | Speculative RAG verifier · TriviaQA and PubHealth | accuracy: -0.19; -1.12 | Helped |
| Add rho_Self-contain to verification scoring (ablation) | Speculative RAG verifier · TriviaQA and PubHealth | accuracy: -2.20; -0.71 | Helped |
| Add rho_Self-reflect to verification scoring (ablation) | Speculative RAG verifier · TriviaQA and PubHealth | accuracy: -1.88; -0.92 | Helped |
| Verify drafts instead of selecting one randomly (ablation) | Speculative RAG · TriviaQA and PubHealth | accuracy: -5.69; -5.37 | Helped |
| Use generated rationale instead of retrieved documents during verification (ablation) | Speculative RAG · TriviaQA and PubHealth | accuracy: 74.08 vs 74.24; 76.09 vs 76.60 | Mixed |
| Add retrieved documents alongside the generated rationale during verification (ablation) | Speculative RAG · TriviaQA and PubHealth | accuracy: +0.08; -0.31 | Mixed |
| Use one draft with all retrieved documents rather than multiple drafts with verification (ablation) | Speculative RAG · TriviaQA and PubHealth | accuracy: 73.23 vs 73.91; 65.25 vs 75.79 | Hurt |
| Train the drafter to generate rationales (ablation) | Speculative RAG drafter · TriviaQA, PubHealth, ARC-Challenge | accuracy: 70.86 vs 74.24; 75.28 vs 76.60; 78.67 vs 80.55 | Helped |
| Increase the number of parallel drafts (ablation) | Speculative RAG · TriviaQA and PubHealth | accuracy | Helped |
Surprising findings
- Gemma-2 2B as drafter performs much worse (67.22 TriviaQA) than Mistral-7B drafter, though the authors say it suggests future promise.
- Increasing the number of documents per draft does not consistently improve performance; the drafter with a single document per draft already beats Mistral-Instruct-7B.
Limitations
- Bridge-type multi-hop questions gain little from multi-perspective sampling because the method does not explicitly identify bridge entities.
- Rationale generation during drafting adds inference cost, relying on parallel inference to offset it.
- Drafter training relies on proprietary LLMs (Gemini-Ultra, GPT-4o) for rationale generation.
Experimental setup
Datasets: Open-Instruct processed data knowledge-intensive datasets (TriviaQA, MuSiQue, PopQA, PubHealth, ARC-Challenge) HotpotQA
Benchmarks: TriviaQA MuSiQue PopQA PubHealth ARC-Challenge HotpotQA
Metrics: accuracy latency (s)
Baselines: Mistral-7B standard RAG Mixtral-8x7B standard RAG Mistral-Instruct-7B standard RAG Mixtral-Instruct-8x7B standard RAG Self-RAG CRAG Self-CRAG SAIL Toolformer Alpaca-7B
Builds on: Speculative Decoding Self-RAG Corrective RAG Contriever
Closest papers by meaning
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.92)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.92)
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.91)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.91)
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.91)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.