Research map / Large language models / Retrieval and knowledge

CRAG - Comprehensive RAG Benchmark

NeurIPS 2024 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation · 21 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CRAG, a 4,409-question factual QA benchmark with mock web and KG APIs for evaluating RAG systems, and benchmarks LLM-only, straightforward RAG, and industry RAG systems.

Problem

Existing RAG and factual QA benchmarks do not capture the diversity, temporal dynamism, entity popularity range, and complex question types of real-world QA, nor do they evaluate web and knowledge graph retrieval together with reliable hallucination-aware scoring.

Approach

A factual QA benchmark spanning five domains and eight question types, with dynamism and popularity labels, paired with up to 50 full HTML web pages per question and mock KG APIs, scored with a truthfulness metric that penalizes hallucinations more than missing answers.

Key results

• LLM-only GPT-4 Turbo reaches 33.5% accuracy, while RAG with Task 3 reaches 43.6% accuracy for GPT-4 Turbo.
• Best industry RAG system (Copilot Pro) achieves 62.6% perfect answers and 50.6% human-eval truthfulness on equal-weighted questions.
• Automatic evaluators reach average F1 of 94.7% (ChatGPT) and 98.9% (Llama 3) against human labels.

Measured effects reported in the paper

Surprising findings

• No straightforward RAG solution achieves truthfulness above 20% because retrieval noise increases hallucinations.
• Over 30% of questions have ground truth present in web retrieval results but omitted from the prompt due to context window limits.

Limitations

• Benchmark is English-only and text-only; multilingual and multimodal questions are future work.
• Industry system comparisons use human evaluation and are not fully comparable to auto-eval results of straightforward solutions.
• Web search recall is estimated with an LLM judge (Llama 3 70B), and only 85% of questions have ground truth in the retrieved pages.

Experimental setup

Datasets: CRAG QA pairs built from KGs and web search results

Benchmarks: CRAG

Metrics: accuracy hallucination rate missing rate truthfulness latency

Baselines: Llama 3 70B Instruct GPT-4 Turbo Llama 2 Chat Mixtral-8x7B-Instruct Falcon 40B FLAN-T5-XXL Copilot Pro Gemini Advanced ChatGPT Plus Perplexity.ai Meta SG

Builds on: Natural Questions TriviaQA MS MARCO QALD-10 FreshLLM

Closest papers by meaning

• Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.87)
• Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.86)
• Chain-of-Retrieval Augmented Generation (NeurIPS 2025 · similarity 0.86)
• HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks (NeurIPS 2025 · similarity 0.86)
• Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking (NeurIPS 2025 · similarity 0.85)
• ChatQA: Surpassing GPT-4 on Conversational QA and RAG (NeurIPS 2024 · similarity 0.85)