Research map / Large language models / Retrieval and knowledge
Chain-of-Retrieval Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains Llama-3.1-8B to iteratively retrieve and reason via rejection-sampled retrieval chains (CoRAG), with test-time decoding strategies that trade tokens for accuracy on multi-hop QA and KILT.
Problem
Single-step RAG retrieves once before generation, so imperfect retrieval limits performance on complex multi-hop queries that require step-by-step information seeking.
Approach
Augment QA-only RAG datasets with automatically generated retrieval chains (sub-queries and sub-answers) via rejection sampling, then fine-tune an open LLM with next-token prediction so it dynamically reformulates queries; scale test-time compute via greedy, best-of-N, and tree search decoding.
Key results
- On multi-hop QA, CoRAG-8B (L=10, best-of-8) reaches 72.5 EM on 2WikiQA and 30.9 EM on MuSiQue, improving over baselines by more than 10 EM points on several datasets.
- On the KILT hidden test set, CoRAG-8B sets new state-of-the-art on all tasks except FEVER (93.1 vs 93.5 for Atlas-11B).
- Retrieval recall at R@10 on MuSiQue rises from 29.0 (E5-large) to 47.1 with CoRAG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoRAG instead of the strongest listed baseline | 2WikiQA · 2WikiQA | EM: 58.0 -> 72.5 | Helped |
| Use CoRAG instead of the strongest listed baseline | HotpotQA · HotpotQA | EM: 50.3 -> 56.3 | Helped |
| Use CoRAG instead of the strongest listed baseline | MuSiQue · MuSiQue | EM: 26.1 -> 30.9 | Helped |
| Use CoRAG instead of the strongest listed baseline | Bamboogle · Bamboogle | EM: 56.0 -> 54.4 | Hurt |
| Increase chain length from L=1 to L=6 with greedy decoding (ablation) | CoRAG-8B · 2WikiQA | EM: 56.5 -> 70.6 | Helped |
| Add iterative rejection-sampling training (ablation) | CoRAG-8B (L=6, greedy) · 2WikiQA | EM: 70.6 -> 72.2 | Helped |
| Add iterative rejection-sampling training (ablation) | CoRAG-8B (L=6, greedy) · HotpotQA | EM: 54.4 -> 53.4 | Hurt |
| Generate retrieval chains by distilling from GPT-4o (ablation) | CoRAG-8B (L=6, greedy) · 2WikiQA | EM: 70.6 -> 75.1 | Helped |
| Generate retrieval chains with Llama-3.2-1B-Instruct instead of the standard generator (ablation) | CoRAG-8B · MuSiQue | EM: 27.7 -> 22.3 | Hurt |
| Use Llama-3.2-3B-Instruct for retrieval-chain generation (ablation) | CoRAG-8B · 2WikiQA | EM: 70.6 -> 69.9 | No clear effect |
| Add chain-of-retrieval at test time with E5-base instead of single retrieval (ablation) | CoRAG with E5-base · MuSiQue | EM: 15.8 -> 26.3 | Helped |
| Add chain-of-retrieval at test time with BM25 instead of single retrieval (ablation) | CoRAG with BM25 · MuSiQue | EM: 14.3 -> 23.5 | Helped |
Surprising findings
- Iterative rejection sampling (a second round using the trained CoRAG model) gives mixed results, with declines on several datasets.
- Llama-3.2-1B as the chain generator yields a noticeable drop, while 3B is close to 8B; scaling rejection sampling compute does not consistently improve performance.
Limitations
- Studies only short, easy-to-verify answer tasks; long-form generation not addressed.
- Test-time token cost counts prompt and generated tokens equally and ignores retrieval costs.
- Bamboogle is only 125 instances, giving high variance; Wikipedia-only retrieval lacks recent knowledge.
Experimental setup
Datasets: 2WikiMultihopQA HotpotQA MuSiQue KILT training set BLINK Wikipedia KILT corpus
Benchmarks: 2WikiMultihopQA HotpotQA Bamboogle MuSiQue KILT
Metrics: exact match F1 Recall@k accuracy R-Precision
Baselines: Llama-3.1-8B-Instruct few-shot GPT-4o few-shot Self-RAG-7B ITER-RETGEN DRAG IterDRAG Search-o1-32B Fine-tuned Llama-8B w/ E5-large KILT-RAG SEAL Atlas-11B RA-DIT 65B
Builds on: FLARE IRCoT ITER-RETGEN Self-RAG Search-o1
Closest papers by meaning
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.89)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.87)
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.86)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.86)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.86)
- One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.