Research map / Large language models / Retrieval and knowledge
LLM-Check: Investigating Detection of Hallucinations in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-Check, training-free hallucination detectors (Hidden Score, Attention Score, and output-token uncertainty measures) that use a single teacher-forced forward pass of the LLM or a substitute open LLM to flag hallucinations in a single response, with large speedups over sampling-based baselines.
Problem
Detecting hallucinations within a single LLM response in white-box and black-box settings, with or without reference documents, without the multi-sample or retrieval overhead of existing detectors.
Approach
Scores computed from a single teacher-forced forward pass: the mean log-determinant of hidden-state covariance (Hidden Score) and the mean log of diagonal attention kernel entries (Attention Score), combined with perplexity, windowed logit entropy, and top-k logit entropy; black-box use relies on an auxiliary open LLM.
Key results
- On FAVA-Annotation with Llama-2-7B, the Attention Score reaches 72.34 AUROC versus 59.03 for INSIDE and 50.30 for Self-Prompt.
- On the SelfCheckGPT dataset, Attention Score gets 80.04 AUC-PR for Llama-2 versus 72.84 for SelfCheck.
- LLM-Check runs with speedups up to 45x and 450x over baselines, with all 32 layers' Attention Score computed in 0.22 s per example on one A5000 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Attention Score instead of INSIDE | Llama-2-7B hallucination detection · FAVA-Annotation | AUROC: 59.03 -> 72.34 | Helped |
| Use Attention Score instead of SelfCheck | Hallucination detection · SelfCheckGPT Dataset | AUC-PR: 72.84 -> 80.04 | Helped |
| Prompt with additional generated responses and compute Attention Scores | LLM-Check · SelfCheckGPT Dataset | Accuracy: 58.91 -> 61.21 | Mixed |
| Use Hidden Score rather than other detection measures | Llama-2-7B white-box detection · RAGTruth | F1 Score: 61.51 | Helped |
| Use Attention Score for black-box detection | LLM-Check · RAGTruth | AUROC: 58.30 | Helped |
| Use output-token entropy rather than Attention Score | Hallucination detection · FAVA Train-split | AUROC: 74.20 -> 69.57; 77.00 -> 71.91 | Helped |
| Combine the proposed scores using Logistic Regression (ablation) | LLM-Check · FAVA-Annotation | detection performance | No clear effect |
| Train classifiers on hidden representations instead of using Attention Score | Llama-2-7B hallucination detection · FAVA-Annotation | AUROC: 55.76, 56.52 versus 72.34 | Hurt |
Surprising findings
- SelfCheckGPT-Prompt performs far worse when run with Llama-2-7B as the judge than the GPT-3-based version reported in the original paper.
- Logistic-regression probes on hidden states from layers 20 and 30 perform worse than the training-free Attention Score on FAVA-Annotation.
Limitations
- Detection true positive rate at low FPR remains limited, with significant room for improvement.
- Requires access to an open-source proxy LLM for black-box settings.
- Simple score combinations via logistic regression gave no gains over the Attention Score alone.
Experimental setup
Datasets: FAVA-Annotation FAVA train split SelfCheckGPT WikiBio dataset RAGTruth Summarization
Benchmarks: FAVA-Annotation SelfCheckGPT WikiBio RAGTruth FAVA train split
Metrics: AUROC accuracy TPR@5%FPR F1 score AUC-PR runtime
Baselines: SelfCheckGPT INSIDE Self-Prompt FAVA Model RAGTruth fine-tuned detector Logit entropy / perplexity Logistic regression probes on hidden states
Builds on: SelfCheckGPT INSIDE Semantic Entropy
Closest papers by meaning
- Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.95)
- HaDeMiF: Hallucination Detection and Mitigation in Large Language Models (ICLR 2025 · similarity 0.95)
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations (NeurIPS 2025 · similarity 0.94)
- ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability (ICLR 2025 · similarity 0.93)
- ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models (NeurIPS 2024 · similarity 0.93)
- Attributive Reasoning for Hallucination Diagnosis of Large Language Models (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.