Research map / Large language models / Language, safety and interpretability
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
Lorenz Kuhn, Yarin Gal, Sebastian Farquhar
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace token-sequence entropy with semantic entropy (entropy over bidirectional-entailment meaning clusters) | OPT-30B uncertainty estimation · TriviaQA | AUROC: 0.828 vs 0.802 for normalised entropy | Improved |
| Replace token-sequence entropy with semantic entropy | OPT-30B uncertainty estimation · CoQA | AUROC: 0.7672 vs 0.7533 for normalised entropy | Improved |
| Use multinomial beam search instead of multinomial sampling to generate answer set (ablation) | Semantic entropy · CoQA | AUROC: 0.735 vs 0.758 | Hurt |
| Sampling temperature 0.5 vs higher temperatures (ablation) | Semantic entropy and normalised entropy · TriviaQA | AUROC | Improved |
| Use margin-probability instead of semantic entropy | OPT-13B uncertainty on CoQA · CoQA | AUROC: AUROC 0.54 | Hurt |
| Use semantic entropy instead of counting distinct semantic answers | OPT-30B uncertainty · TriviaQA | AUROC: 0.83 vs 0.79 | Improved |
| Use semantic entropy instead of counting distinct semantic answers | OPT-30B uncertainty · CoQA | AUROC: 0.77 vs 0.66 | Improved |
| Semantic clustering (vs. token-level entropy) at high temperature T=1.5 (ablation) | Semantic entropy · TriviaQA | semantic equivalence accuracy: 61% vs over 92% at lower temperatures | Hurt |
Closest papers by meaning
- Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.91)
- Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.91)
- Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models (NeurIPS 2025 · similarity 0.91)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.90)
- Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.89)
- Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
