Research map / Large language models / Language, safety and interpretability
Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kernel Language Entropy (KLE), which computes the von Neumann entropy of NLI-derived semantic unit-trace kernels to give fine-grained uncertainty estimates for LLM outputs, generalizing semantic entropy.
Problem
Estimating semantic uncertainty of free-form LLM generations is hard because semantic entropy only uses hard equivalence clusters and ignores graded semantic similarity between answers, which hurts hallucination detection.
Approach
Encode semantic similarity between generations (or clusters) as a unit-trace PSD kernel built from NLI-derived weighted graphs (heat or Matérn graph kernels), and measure uncertainty as the von Neumann entropy of that kernel; this provably generalizes semantic entropy.
Key results
- KLE(K_heat) achieves the best overall performance, winning across the 60 model-dataset scenarios against baselines including semantic entropy and P(True).
- On Llama 2 70B Chat, KLE(K_heat) reaches AUROC 0.84 on TriviaQA vs 0.77 for SE and 0.79 for P(True)-level baselines; on Falcon 40B Instruct KLE(K_heat) reaches 0.92 AUROC on BioASQ vs 0.85 for SE.
- The paper proves KLE (with block-diagonal kernels) recovers semantic entropy exactly for any semantic clustering, so it is strictly more general.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KLE(K_heat) instead of semantic entropy | Llama 2 70B Chat · Trivia QA | AUROC: 0.77 ± 0.03 -> 0.84 ± 0.03 | Helped |
| Use KLE(K_heat) instead of semantic entropy | Falcon 40B Instruct · BioASQ | AUROC: 0.85 ± 0.02 -> 0.92 ± 0.01 | Helped |
| Use KLE(K_heat) instead of semantic entropy | Falcon 40B Instruct · NQ | AUROC: 0.78 ± 0.03 -> 0.76 ± 0.03 | Hurt |
| Use KLE(K_heat) instead of semantic entropy | Falcon 40B Instruct · NQ | AUARC: 0.43 ± 0.03 -> 0.42 ± 0.03 | Hurt |
| Use KLE(K_heat) instead of KLE(K_full) (ablation) | KLE · Llama 2 70B Chat, Trivia QA | AUROC: 0.80 ± 0.03 -> 0.84 ± 0.03 | Helped |
| Use KLE(K_heat) instead of KLE(K_full) (ablation) | KLE · Falcon 40B Instruct, SVAMP | AUROC: 0.69 ± 0.03 -> 0.77 ± 0.02 | Helped |
| Use default kernel hyperparameters selected from entropy convergence plots rather than validation-set selection (ablation) | KLE · 60 experimental scenarios | AUROC | No clear effect |
| Use heat kernel rather than other evaluated kernel design choices (ablation) | KLE · 48 experiments | overall performance | Helped |
| Use KLE rather than existing uncertainty methods | Instruction-tuned models · 60 experimental scenarios | AUROC | Helped |
| Use KLE rather than existing uncertainty methods | Non-instruction-tuned models · 60 experimental scenarios | AUROC | Mixed |
| Include semantic probabilities by using variations of K_full (ablation) | KLE on non-instruction-tuned models · Non-instruction-tuned scenarios | uncertainty quantification performance | Helped |
Surprising findings
- Token-likelihood-based methods perform better on non-instruction-tuned models, while KLE's gains are largest for instruction-tuned models.
Limitations
- Requires multiple sampled generations per query, increasing generation cost.
- Only NLI-based semantic kernels are studied; embedding-based or other kernels are left for future work.
- Evaluated only on NLG QA and math datasets, not on code generation or other applications.
Experimental setup
Datasets: TriviaQA SQuAD BioASQ Natural Questions SVAMP
Benchmarks: TriviaQA SQuAD BioASQ Natural Questions SVAMP
Metrics: AUROC AUARC
Baselines: semantic entropy discrete semantic entropy token predictive entropy embedding regression P(True)
Builds on: semantic entropy discrete semantic entropy P(True) DeBERTa-Large-MNLI
Closest papers by meaning
- Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models (NeurIPS 2025 · similarity 0.92)
- Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space (NeurIPS 2024 · similarity 0.91)
- Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.91)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.91)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.91)
- Integrating Local and Global Entropy for Uncertainty Quantification in LLMs (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.