Research map / Large language models / Language, safety and interpretability

To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a mutual-information metric computed from iterative prompting over an LLM's previous responses to detect epistemic uncertainty and hallucinations in single- and multi-answer queries.

Problem

Existing uncertainty quantification and hallucination detection methods assume a single correct answer, so they fail on multi-answer queries where high output entropy reflects aleatoric rather than epistemic uncertainty.

Approach

Build a pseudo joint distribution over multiple responses by iteratively prompting the LLM with its previous answers; under a ground-truth independence assumption, the mutual information of this joint is a computable lower bound on KL divergence from the truth, so high MI signals epistemic uncertainty and hallucination. A finite-sample MI estimator with a high-probability bound is used with calibr

Key results

• On TriviaQA and AmbigQA, the M.I. method performs nearly identically to semantic entropy and outperforms the greedy-probability and self-verification baselines.
• On mixed single/multi-label datasets (TriviaQA+WordNet, AmbigQA+WordNet), M.I. significantly outperforms semantic entropy, which achieves nearly zero recall on high-entropy queries while M.I. makes few mistakes there.
• Iterative prompting with repeated incorrect responses can collapse the probability of the correct answer for single-label queries with high epistemic uncertainty (e.g., "What is the national instrument of Ireland?"), while multi-label correct answers stay probable.

Measured effects reported in the paper

Surprising findings

• Repeating an incorrect answer in the prompt can force the LLM to output it, with the correct answer's probability falling to near zero in high-epistemic-uncertainty single-answer queries.
• Entropy-based semantic entropy does well on most single-label benchmarks, so the proposed method's advantage appears only when multi-label queries are mixed in.

Limitations

• Requires many sampled responses and repeated prompting, with costs that scale with sample size k.
• Assumes ground-truth independence of responses given valid answers, which may fail for partial answers or prompts that redefine the problem.
• Experiments are limited to closed-book QA with a single proprietary model family.

Experimental setup

Datasets: TriviaQA AmbigQA WordNet (synthesized multi-label queries)

Benchmarks: TriviaQA AmbigQA TriviaQA+WordNet AmbigQA+WordNet

Metrics: precision-recall curve error rate recall

Baselines: probability of greedy response semantic entropy self-verification

Builds on: semantic entropy self-verification (P(True)) epinets

Closest papers by meaning

• Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.90)
• Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.90)
• Position: Uncertainty Quantification Needs Reassessment for Large Language Model Agents (ICML 2025 · similarity 0.89)
• FLUE: Streamlined Uncertainty Estimation for Large Language Models (AAAI 2025 · similarity 0.89)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.88)
• Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.88)