AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Lorenz Kuhn, Yarin Gal, Sebastian Farquhar

ICLR 2023 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 52 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace token-sequence entropy with semantic entropy (entropy over bidirectional-entailment meaning clusters)OPT-30B uncertainty estimation · TriviaQAAUROC: 0.828 vs 0.802 for normalised entropyImproved
Replace token-sequence entropy with semantic entropyOPT-30B uncertainty estimation · CoQAAUROC: 0.7672 vs 0.7533 for normalised entropyImproved
Use multinomial beam search instead of multinomial sampling to generate answer set (ablation)Semantic entropy · CoQAAUROC: 0.735 vs 0.758Hurt
Sampling temperature 0.5 vs higher temperatures (ablation)Semantic entropy and normalised entropy · TriviaQAAUROCImproved
Use margin-probability instead of semantic entropyOPT-13B uncertainty on CoQA · CoQAAUROC: AUROC 0.54Hurt
Use semantic entropy instead of counting distinct semantic answersOPT-30B uncertainty · TriviaQAAUROC: 0.83 vs 0.79Improved
Use semantic entropy instead of counting distinct semantic answersOPT-30B uncertainty · CoQAAUROC: 0.77 vs 0.66Improved
Semantic clustering (vs. token-level entropy) at high temperature T=1.5 (ablation)Semantic entropy · TriviaQAsemantic equivalence accuracy: 61% vs over 92% at lower temperaturesHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy