Research map / Large language models / Retrieval and knowledge
MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations
Ernests Lavrinovics, Russa Biswas, Katja Hose, Johannes Bjerva
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KG paths as in-context knowledge (KG-RAG) instead of vanilla QA | GPT-4o Mini · MultiHal NLI, all languages | entailment (%): 40% -> 76% | Improved |
| add KG paths as in-context knowledge | Gemini 2.0 Flash · MultiHal NLI, all languages | entailment (%): 52% -> 68% | Improved |
| add KG paths as in-context knowledge | Llama 3.3 70B Instruct · MultiHal NLI, all languages | entailment (%): 45% -> 71% | Improved |
| add KG paths as in-context knowledge | GPT-4o Mini · MultiHal English, HHEM-2.1 | hallucinated (%): 53% -> 11% | Improved |
| add KG paths as in-context knowledge | Gemini 2.0 Flash · MultiHal English, HHEM-2.1 | hallucinated (%): 42% -> 13% | Improved |
| add KG paths as in-context knowledge | Llama 3.3 70B Instruct · MultiHal English, HHEM-2.1 | hallucinated (%): 45% -> 12% | Improved |
| restrict KG paths to quality rating 4 only versus vanilla QA (ablation) | GPT-4o Mini · MultiHal semantic similarity | semantic similarity: 0.4 -> 0.52 | Improved |
| use GPT-4o Mini instead of Gemini 2.0 Flash as KG-path judge (ablation) | KG path filtering · English subset, KG-RAG | correlation of path score with semantic score: 0.430 -> 0.485 | Improved |
| use GPT-4o Mini instead of Gemini 2.0 Flash as KG-path judge (ablation) | KG path filtering · English subset, KG-RAG | semantic score: 0.529 -> 0.513 | No clear effect |
| separate KG path labels with semicolons before translation (ablation) | NLLB-200 translation · translation of KG paths | translation quality | Improved |
| add KG paths for suggestive, temporal or reasoning-heavy subsets | KG-RAG · TruthfulQA, HaluBench, Felm subsets | semantic similarity | Mixed |
Closest papers by meaning
- Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling (AAAI 2025 · similarity 0.92)
- In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.92)
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.92)
- Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detection (AAAI 2025 · similarity 0.92)
- Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting (AAAI 2024 · similarity 0.92)
- HaDeMiF: Hallucination Detection and Mitigation in Large Language Models (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
