Research map / Large language models / Retrieval and knowledge
RAGGED: Towards Informed Design of Scalable and Stable RAG Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RAGGED, a framework and two metrics (RAG Stability Score and RAG Scalability Coefficient) for evaluating how retriever-reader configurations and retrieval depth affect RAG performance across NQ, HotpotQA, and BioASQ.
Problem
RAG effectiveness depends heavily on system configuration (retrieval depth, retriever, reader), and improper settings can make RAG worse than closed-book generation; prior studies disagree on how increasing retrieval depth affects performance.
Approach
Evaluates readers on naturally retrieved top-k contexts rather than curated or injected distractors, and proposes RSS (worst-case performance in a window around optimal k) and RSC (area under the F1-vs-k curve until gains stop) to quantify stability and scalability; finds reader noise robustness, not retriever quality, drives the trends.
Key results
- FlanT5 achieves an RSS of 0.99 on NQ with ColBERT, while LLaMA2 models peak at small k and decline steadily.
- ColBERT improves NQ recall@k by 21.3 points over BM25 but yields only +5.2 F1 reader gain on average.
- In BioASQ, a +0.7 recall@k improvement from ColBERT yields +2.08 F1 reader gain, suggesting specialized domains benefit more from retrieval gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ColBERT retrieval instead of no-context generation | FlanT5 · NQ | F1 difference from no-context: +28.9 | Helped |
| Use ColBERT retrieval instead of no-context generation | LLaMa3 70B · NQ | F1 difference from no-context: -24.9 | Hurt |
| Use ColBERT retrieval instead of no-context generation | GPT-3.5 · NQ | F1 difference from no-context: 1.1; -8.0 | Mixed |
| Increase retrieval depth from k=1 to k=5 | FlanT5 · CRAG | F1: 0.19 -> 0.17 | Hurt |
| Increase retrieval depth from k=1 to k=5 | LLaMa2 · CRAG | F1: 0.20 -> 0.23 | Helped |
| Replace BM25 with ColBERT | RAG readers · NQ | Average F1 difference across k: +5.2 | Helped |
| Replace BM25 with ColBERT | RAG readers · HotpotQA | Average F1 difference across k: +1.9 | Helped |
| Replace BM25 with ColBERT | RAG readers · BioASQ | Average F1 difference across k: +1.5 | Helped |
| Increase retrieval depth beyond the model’s peak | LLaMa and Claude readers · RAG tasks | F1 | Hurt |
| Increase retrieval depth until the performance plateau | Flan and GPT-3.5 readers · RAG tasks | F1 | Helped |
| Apply explicit relevance prompting | FlanT5 and LLaMa2 7B · NQ | F1 | Mixed |
| Apply explicit relevance prompting | RAG readers · BioASQ | F1 | Hurt |
Surprising findings
- Stronger retriever ColBERT performs worse than BM25 at large k for peak-then-decline readers such as LLaMA and Claude-3-Haiku.
- Several readers (e.g., LLaMA3 8B/70B, Claude-3-Haiku on NQ) fall below closed-book generation under RAG, while FlanT5 improves even without gold passages.
Limitations
- RSS uses a symmetric window and does not capture directional asymmetry between adding noise and omitting content.
- Closed-source model internals are unknown, so architectural explanations for reader behavior are hypothetical.
- CRAG results are preliminary with only two readers and five k values.
Experimental setup
Datasets: Natural Questions (NQ) HotpotQA BioASQ Task 11B KILT Wikipedia paragraphs corpus PubMed Annual Baseline 2023 CRAG
Benchmarks: NQ HotpotQA BioASQ Task 11B CRAG
Metrics: unigram F1 recall@k RAG Stability Score (RSS) RAG Scalability Coefficient (RSC)
Baselines: GPT-3.5-turbo Claude-3-Haiku FlanT5-XXL Flan-UL2 LLaMA2 LLaMA3 GPT-4o
Builds on: BM25 ColBERT Contriever KILT Prometheus
Closest papers by meaning
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.92)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.91)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.91)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.91)
- Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems (ICLR 2025 · similarity 0.91)
- Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.