Research map / Probabilistic and scientific modeling / Variational and generative sampling

Estimating the Hallucination Rate of Generative AI

NeurIPS 2024 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Score-based generative modeling analysis · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the posterior hallucination rate (PHR), a Bayesian definition of hallucination for in-context learning, and an estimator that uses only sampled predictive completions and response log-probabilities from a conditional generative model.

Problem

Estimating how often an in-context learning (ICL) prediction from a conditional generative model such as an LLM will be a hallucination, without access to the true data-generating mechanism or labeled evaluation data.

Approach

Define hallucination as a response with low likelihood under the latent mechanism implied by a Bayesian view of ICL, then estimate the posterior expectation of that rate by predictive resampling: sample future context pairs from the CGM, then check whether sampled responses fall in the tails of the log-likelihood distribution given the extended context, using Doob's theorem to justify the approxim

Key results

• On synthetic regression, the PHR closely tracks the true hallucination rate across context lengths but tends to underestimate it, with lower R^2 as context length grows.
• On natural language ICL with Llama-2-7B, PHR (epsilon=0.05) closely follows the model hallucination rate across SST2, Subjectivity, and AG News as the number of in-context examples increases.
• With epsilon=0.75, PHR tracks the empirical error rate for SST2, Subjectivity, and AG News but diverges for Medical QP, RTE, and WNLI, where Llama-2-7B performs near random.

Measured effects reported in the paper

Surprising findings

• PHR fails to match error rate on RTE, WNLI and Medical QP because Llama-2-7B is near random guessing on those tasks, so the posterior predictive is poorly approximated.

Limitations

• Relies on the de Finetti/exchangeability assumption and on the CGM being a faithful approximation of the ICL predictive distribution.
• PHR underestimates the true hallucination rate on synthetic regression, especially at large context lengths.
• Captures only hallucinations from insufficient context (special epistemic uncertainty), not those from lack of model capability (general epistemic uncertainty).

Experimental setup

Datasets: SST2 Subjectivity AG News Medical QP RTE WNLI synthetic random MLP regression functions

Benchmarks: SST2 Subjectivity AG News Medical QP RTE WNLI synthetic regression

Metrics: MSE regression coefficient beta p-value R^2 empirical error rate agreement

Baselines: Model hallucination rate (MHR) empirical error rate true hallucination rate (THR)

Builds on: Martingale posterior Doob's theorem Conditional Neural Process Llama 2

Closest papers by meaning

• No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models (ICLR 2025 · similarity 0.91)
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.89)
• Attributive Reasoning for Hallucination Diagnosis of Large Language Models (AAAI 2025 · similarity 0.89)
• Entropy Distribution as a Fingerprint for Hallucinations in Generative Models (NeurIPS 2026 · similarity 0.89)
• Sample Transform Cost-Based Training-Free Hallucination Detector for Large Language Models (NeurIPS 2026 · similarity 0.89)
• Multi-Modal Hallucination Control by Visual Information Grounding (CVPR 2024 · similarity 0.89)