Research map / Large language models / Language, safety and interpretability
S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs
Yawei Li, Jiazheng Li, David Rügamer, Bernd Bischl, Yulan He, Mina Rezaei
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S-EDL, a method that elicits self-evidence from sequence likelihoods to semantically calibrate LLM open-ended generation, where many token sequences share one meaning.
Problem
Calibrating LLMs in open-ended generation is hard because many distinct token sequences express the same meaning, while prior calibration methods target fixed-set classification confidence or binary correctness of individual strings.
Approach
Elicits self-evidence from sequence likelihoods so that confidence is computed at the level of semantic outcomes (meaning clusters) rather than individual strings or fixed label sets; the abstract is truncated so the precise mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with S-EDL instead of plain SFT | Qwen3-8B-Base · Average of 4 datasets | smECE: 16.5 -> 10.5 | Improved |
| Fine-tune with S-EDL instead of plain SFT | Qwen3-8B-Base · Average of 4 datasets | Pass@1: 60.9 -> 70.5 | Improved |
| Add label smoothing to SFT | MiMo-7B-Base · Average of 4 datasets | Pass@1: 56.2 -> 53.5 | Hurt |
| Add label smoothing to SFT | MiMo-7B-Base · Average of 4 datasets | smECE: 12.1 -> 11.9 | No clear effect |
| Use binary-reward Dr.GRPO instead of token-level RCFT | Ministral3-8B-Base · Average of 4 datasets | smECE: 10.9 -> 11.4 | Hurt |
| Use S-EDL instead of Dr.GRPO (cross-domain) | Qwen3-8B-Base · MATH-trained, cross-domain average | smECE: 10.6 -> 5.9 | Improved |
| Use S-EDL instead of Dr.GRPO (cross-domain) | MiMo-7B-Base · MATH-trained, cross-domain average | smECE: 12.3 -> 12.2 | No clear effect |
| Add LS, RCFT or BLoB to SFT (cross-domain) | Qwen3-8B-Base and MiMo-7B-Base · MATH-trained, cross-domain average | smECE | Hurt |
| Vary EDL loss weight over four values (ablation) | S-EDL · Ablation of loss weight | Pass@1, S-Acc, smECE | Improved |
| Raise beta from 1.2 to 1.5 (ablation) | S-EDL · Ablation of beta | smECE | Hurt |
| Set beta to 1.01 (ablation) | S-EDL · Ablation of beta | Pass@1, S-Acc, smECE | Hurt |
| Increase N_train from 4 to 32 (ablation) | S-EDL · Ablation of N_train | Pass@1, S-Acc, smECE | Improved |
Limitations
- Only the abstract was available; full method and results are unknown
Experimental setup
Builds on: Evidential deep learning
Closest papers by meaning
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.90)
- Calibrating LLMs with Semantic-level Reward (NeurIPS 2026 · similarity 0.88)
- Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization (NeurIPS 2025 · similarity 0.88)
- Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.88)
- Calibrating LLMs with Information-Theoretic Evidential Deep Learning (ICLR 2025 · similarity 0.88)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
