Research map / Large language models / Language, safety and interpretability
Calibrating Verbalized Confidence with Self-Generated Distractors
Victor Wang, Elias Stengel-Eskin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Normalize verbalized confidence over self-generated distractors and combine with self-consistency (DiNCo) | LLM verbalized confidence · TriviaQA and SimpleQA | ECE: avg ECE gain 0.077 (TriviaQA), 0.092 (SimpleQA) over best baseline | Improved |
| Add self-consistency to NVC (DiNCo vs NVC alone) (ablation) | DiNCo · SimpleQA GPT-4.1 | AUC: 0.729 -> 0.786 | Improved |
| Add distractor normalization to verbalized confidence (NVC vs VC) (ablation) | Qwen3-8B · TriviaQA | ECE: 0.240 -> 0.171 | Improved |
| Add distractor normalization on Qwen3-1.7B (NVC vs DiNCo) (ablation) | DiNCo · TriviaQA Qwen3-1.7B | ECE: 0.084 (NVC) vs 0.177 (DiNCo) | Hurt |
| Scale self-consistency to 100 samples to match DiNCo (ablation) | Self-consistency · TriviaQA Qwen3-8B | calibration | No clear effect |
| Remove NLI-based weighting of distractors (ablation) | DiNCo · TriviaQA Qwen3-8B | calibration | Hurt |
| Black-box setting (no logit access) vs logit access (ablation) | DiNCo · SimpleQA GPT-4.1 | AUC: 0.786 -> 0.605 | Hurt |
| Apply DiNCo to long-form biography generation | Qwen3-8B · FactScore | ECE: 0.162 (SC) -> 0.076 | Improved |
| Apply DiNCo vs SC on FactScore correlation | DiNCo · FactScore | Pearson / Spearman: +0.072 / +0.074 avg over SC | Improved |
| Use DiNCo instead of MSP on GPT-4.1 SimpleQA AUC | DiNCo · SimpleQA GPT-4.1 | AUC: 0.800 (MSP) vs 0.786 | Hurt |
| Use DiNCo vs Gemma-3-4B FactScore ECE of NVC (ablation) | DiNCo · FactScore Gemma-3-4B-IT | ECE: 0.123 (NVC) vs 0.172 | Hurt |
Closest papers by meaning
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.90)
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.90)
- SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.89)
- ConfTuner: Training Large Language Models to Express Their Confidence Verbally (NeurIPS 2025 · similarity 0.89)
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.88)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
