Research map / Large language models / Retrieval and knowledge

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that hallucination labels for open-domain QA benchmarks depend on whether the labeler applies reference faithfulness or factual correctness, using 900 human-labeled QA pairs to audit lexical metrics, NLI, and seven LLM judges.

Problem

Automated hallucination labels in open-domain QA benchmarks often compare answers to short reference answers, conflating reference faithfulness with factual correctness, which may bias evaluation of hallucination detectors.

Approach

Formalizes the gap between reference faithfulness (unsupported-by-reference counts as hallucination) and factual correctness (only contradictions or false specific claims count), and shows that switching an LLM judge prompt from faithfulness to factual-correctness criterion reduces false positives and improves human agreement.

Key results

• GPT-5-mini with factual-correctness prompt p2 raised kappa vs human labels from 0.194 to 0.626 on Mistral-7B outputs and from 0.530 to 0.756 on Llama-3-8B outputs relative to faithfulness prompt p1.
• Strict T5-11B NLI labels were false-positive dominant, with FP share 98.3-99.4% of errors and kappa 0.025-0.053.
• Inter-annotator agreement on a 300-item subset was Cohen's kappa 0.808 (95% CI 0.729-0.879).

Surprising findings

• Frontier judges differ strongly under the same faithfulness prompt: GPT-5.4 p1 kappa 0.021 on Mistral outputs versus Claude Opus 4.7 0.693.
• A p1 variant with p2-style structure gave no detectable improvement, indicating the criterion rather than prompt structure drives the gain.

Limitations

• Single primary annotator labels for the 900 items; no consensus adjudication.
• Only English, short-reference open-domain QA with three small open-weight generators.
• Answer-level binary labels cannot capture severity or partial correctness.

Experimental setup

Datasets: TriviaQA HotpotQA TruthfulQA

Benchmarks: 900-pair human-labeled QA set (TriviaQA, HotpotQA, TruthfulQA)

Metrics: Cohen's kappa positive-label bias false positive rate false negative rate FP % McNemar's test

Baselines: ROUGE-L BERTScore NLI (T5-11B) METEOR BARTScore

Builds on: TRUE TriviaQA HotpotQA TruthfulQA

Closest papers by meaning

• Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models (AAAI 2026 · similarity 0.91)
• FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.91)
• ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models (NeurIPS 2024 · similarity 0.91)
• PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset (CVPR 2025 · similarity 0.90)
• Evaluating Image Hallucination in Text-to-Image Generation with Question-Answering (AAAI 2025 · similarity 0.90)
• In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.89)