Research map / Large language models / Retrieval and knowledge
RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAGChecker, a claim-level diagnostic evaluation framework giving overall, retriever, and generator metrics for RAG systems, validated against human preferences on 280 pairwise instances and used to evaluate 8 RAG systems on a 4,162-query benchmark across 10 domains.
Problem
Evaluating retrieval-augmented generation systems is hard because RAG is modular, long-form responses are poorly scored by existing metrics, and existing metrics' reliability against human judgment is under-explored.
Approach
Decompose responses and ground-truth answers into claims checked by entailment (via RefChecker with Llama3-70B) against retrieved chunks and the ground truth, yielding overall precision/recall/F1 plus retriever (claim recall, context precision) and generator (faithfulness, noise sensitivity, hallucination, self-knowledge, context utilization) diagnostics.
Key results
- RAGChecker achieves the highest correlation with human preference on correctness (Pearson 49.66), completeness (60.67), and overall assessment (61.93), outperforming the best baseline RAGAS Answer Similarity (41.07, 53.16, 48.31).
- Across 8 RAG systems, E5-Mistral_GPT-4 achieves the best averaged F1 of 52.7, with GPT-4 generator consistently ahead of open-source generators in context utilization and noise sensitivity.
- Increasing top-k from 5 to 20 raises claim recall from 61.5 to 77.6 and faithfulness from 88.1 to 92.2, but also increases noise sensitivity from 34.0 to 35.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BM25 with E5-Mistral while holding the generator fixed | RAG systems · RAG benchmark, 10-domain average | F1 | Helped |
| Use Llama3-70B instead of Llama3-8B with the same retriever | RAG systems · RAG benchmark, 10-domain average | F1 | Helped |
| Use RAGChecker instead of RAGAS Answer Similarity for human-preference assessment | RAG evaluation · Human preference, overall assessment | Pearson correlation: 48.31 -> 61.93 | Helped |
| Increase retrieved chunks from k=5 to k=20 (ablation) | RAG systems · RAG benchmark, varied top-k | F1: 51.7→53.4 | Helped |
| Increase retrieved chunks from k=5 to k=20 (ablation) | RAG systems · RAG benchmark, varied top-k | noise sensitivity: 34.0→35.4 | Hurt |
| Increase chunk size from 150 to 300 (ablation) | RAG systems · RAG benchmark, varied chunk size | F1: 52.6→53.4 | Helped |
| Increase chunk size from 150 to 300 (ablation) | RAG systems · RAG benchmark, varied chunk size | noise sensitivity: 34.5→35.4 | Hurt |
| Add explicit prompt requirements for faithfulness, context utilization, and lower noise sensitivity (ablation) | RAG generators · RAG benchmark, optimized generation prompt | context utilization: 59.2→63.7 | Helped |
| Add explicit prompt requirements for faithfulness, context utilization, and lower noise sensitivity (ablation) | RAG generators · RAG benchmark, optimized generation prompt | noise sensitivity: 35.4→38.1 | Hurt |
| Add explicit prompt requirements (ablation) | GPT-4 RAG systems · RAG benchmark, optimized generation prompt | F1 | Helped |
| Add explicit prompt requirements (ablation) | Llama3 RAG systems · RAG benchmark, optimized generation prompt | F1 | No clear effect |
| Increase chunk overlap ratio (ablation) | RAG systems · RAG benchmark, varied chunk overlap | claim recall: 77.8→78.1 | No clear effect |
Surprising findings
- Open-source generators are faithful to context but blindly trust it, becoming more sensitive to noise as retrieval improves.
- Relevant noise sensitivity is consistently higher than irrelevant noise sensitivity, indicating generators trust relevant chunks as a whole.
Limitations
- Retriever diagnostic metrics are less insightful than generator metrics.
- Neutral and contradiction results from RefChecker are treated equally in the metrics.
- Benchmark is English-only and repurposed from existing text-only datasets.
Experimental setup
Datasets: RobustQA BioASQ FiQA LoTTE ClapNQ NovelQA KIWI NaturalQuestions
Benchmarks: RAGChecker benchmark (4,162 queries, 10 domains) ClapNQ NovelQA KIWI RobustQA
Metrics: precision recall F1 claim recall context precision faithfulness context utilization relevant noise sensitivity irrelevant noise sensitivity hallucination
Baselines: BLEU ROUGE-L BERTScore TruLens RAGAS ARES CRUD-RAG
Builds on: RefChecker RAGAS TruLens ARES
Closest papers by meaning
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.91)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.90)
- SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction (ICLR 2025 · similarity 0.88)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.88)
- Reliable Decision‑Making via Calibration‑Oriented Retrieval‑Augmented Generation (NeurIPS 2025 · similarity 0.88)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.