Research map / Large language models / Retrieval and knowledge
Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits 16 KGQA datasets (finding ~57% average factual correctness), then proposes KGQAGen, an LLM-in-the-loop Wikidata-grounded pipeline with SPARQL verification, and releases KGQAGen-10k benchmark evaluated on pure LLMs and KG-RAG systems.
Problem
Popular KGQA benchmarks such as WebQSP and CWQ contain incorrect or outdated ground-truth answers, ambiguous or trivial questions, and rigid exact-match evaluation, undermining reliable evaluation of KG-RAG systems.
Approach
An LLM-guided iterative subgraph expansion from seed entities over Wikidata generates multi-hop questions with supporting subgraphs and SPARQL queries; answers are verified by executing the SPARQL query against the KG, with LLM-based query revision, and evaluated with an LLM-assisted semantic match fallback.
Key results
- Manual audit of 16 KGQA datasets finds average correctness of 57%, with WebQSP at 52% and CWQ at 49.33%.
- Manual inspection of 300 KGQAGen-10k samples finds 96.3% factual correctness.
- On KGQAGen-10k, GPT-4.1 reaches 56.96% LASM accuracy and PoG reaches 60.18%, while GPT-4o with supporting subgraph reaches 84.89%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LASM instead of exact match for evaluation | All evaluated models · KGQAGen-10k test | Accuracy: 32.34 -> 42.49 | Helped |
| Add KG retrieval with RoG | LLaMA2-7B · KGQAGen-10k test | Accuracy (EM): 6.88 -> 20.10 | Helped |
| Add the supporting subgraph as input | GPT-4o · KGQAGen-10k test | Accuracy (LASM): 54.21 -> 84.89 | Helped |
| Add the supporting subgraph as input | LLaMA2-7B · KGQAGen-10k test | Accuracy (LASM): 12.32 -> 73.79 | Helped |
| Use PoG instead of the GPT-4o question-only baseline | GPT-4o · KGQAGen-10k test | Accuracy (LASM): 54.21 -> 60.18 | Helped |
| Use LLM-guided subgraph expansion and SPARQL-based verification (ablation) | KGQAGen · KGQAGen-10k manual audit | factual accuracy | Helped |
Surprising findings
- FreeBaseQA has 98.67% correctness but GPT-4o answers 90.39% of its questions without KG access, suggesting it is too simple.
- LASM semantic matching consistently raises reported scores over exact match across all models.
Limitations
- Relies on the accuracy and coverage of the underlying Wikidata KG.
- Pipeline depends on LLM capabilities and prompt design, so question quality may vary with LLM variability.
- Conservative SPARQL-based filtering discards many generated instances (15,451 generated to 10,787 kept).
Experimental setup
Datasets: WebQSP CWQ Wikidata
Benchmarks: KGQAGen-10k WebQSP CWQ
Metrics: Accuracy Hit@1 F1 Precision Recall exact match LLM-assisted semantic match
Baselines: RoG GCR ToG PoG GPT-4o GPT-4.1 GPT-4 DeepSeek-Chat GPT-4o-mini LLaMA-3.1-8B-Instruct LLaMA2-7B Mistral-7B-Instruct-v0.2
Builds on: Reasoning-on-Graph Graph-Constrained Reasoning Think-on-Graph Path-of-Graph DyVal
Closest papers by meaning
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.90)
- KG-Guard: Graph-Based Hallucination Detection for Knowledge Base Question Answering (NeurIPS 2026 · similarity 0.90)
- Question Calibration and Multi-Hop Modeling for Temporal Question Answering (AAAI 2024 · similarity 0.89)
- EPERM: An Evidence Path Enhanced Reasoning Model for Knowledge Graph Question and Answering (AAAI 2025 · similarity 0.88)
- Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering (AAAI 2025 · similarity 0.88)
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.