Research map / Large language models / Retrieval and knowledge
Benchmarking Large Language Models in Retrieval-Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RGB, a bilingual (English/Chinese) news-based benchmark testing four RAG abilities (noise robustness, negative rejection, information integration, counterfactual robustness) and evaluates six LLMs on it.
Problem
Existing work lacks rigorous evaluation of how retrieval-augmented generation affects different LLMs, making it hard to identify RAG capability bottlenecks such as handling noisy, unanswerable, multi-document, or counterfactual retrieved content.
Approach
A benchmark built from latest news (to avoid parametric-knowledge bias) with search-engine-retrieved documents, split into four testbeds that each isolate one required RAG ability, with metrics tailored to each (accuracy, rejection rate, error detection and correction rates).
Key results
- ChatGPT accuracy on noise robustness drops from 96.33% at noise ratio 0 to 76.00% at 0.8 in English.
- Highest negative rejection rate is only 45% (English, ChatGPT judged) and 43.33% (Chinese), showing LLMs are easily misled by noisy documents.
- Information integration accuracy without noise peaks at 60% (English) and 67% (Chinese), falling to 43% and 55% with noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the noise ratio in retrieved documents from 0 to 0.8 | ChatGPT · RGB noise robustness English | accuracy (%): 96.33 -> 76.00 | Hurt |
| Increase the noise ratio in retrieved documents from 0 to 0.8 | ChatGLM2-6B · RGB noise robustness English | accuracy (%): 91.33 -> 57.33 | Hurt |
| Increase noise in retrieved documents | LLMs using RAG · RGB information integration, English and Chinese | accuracy (%) | Hurt |
| Provide counterfactual documents instead of no documents | ChatGPT-zh · RGB counterfactual robustness Chinese | accuracy (%): 91 -> 17 | Hurt |
| Provide counterfactual documents instead of no documents | Qwen-7B-Chat-zh · RGB counterfactual robustness Chinese | accuracy (%): 77 -> 12 | Hurt |
| Provide counterfactual documents instead of no documents | ChatGPT-en · RGB counterfactual robustness English | accuracy (%): 89 -> 9 | Hurt |
| Use retrieved documents with increasing noise | LLMs using RAG · RGB noise robustness | accuracy (%) | Hurt |
| Ask complex questions requiring information from multiple documents | LLMs using RAG · RGB information integration | accuracy (%) | Hurt |
Surprising findings
- LLMs often fail to follow rejection instructions: exact-match rejection rates are much lower than ChatGPT-judged rates.
- Most LLMs struggle to answer internal-knowledge questions, and models rarely detect or correct factual errors in provided documents (error detection rate at most 8%).
Limitations
- Negative documents and noise are constructed from search results and sampled by ratio, so noise realism is limited to the retrieval pipeline used.
- Exact-match accuracy and rejection metrics are brittle, and ChatGPT judging is used as a complementary evaluator.
- Only six LLMs evaluated, mostly 6-7B open models plus gpt-3.5-turbo, and the counterfactual testbed uses only a subset of models passing a 70% threshold.
Experimental setup
Datasets: latest news articles (collected for QA generation) Google search API retrieved web pages
Benchmarks: RGB
Metrics: accuracy rejection rate error detection rate error correction rate
Baselines: ChatGPT ChatGLM-6B ChatGLM2-6B Vicuna-7B-v1.3 Qwen-7B-Chat BELLE-7B-2M
Builds on: ChatGPT retrieval plugin Google search API
Closest papers by meaning
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.92)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.91)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.90)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.90)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.90)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.