Research map / Large language models / Language, safety and interpretability
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a black-box framework (prompting, sampling, aggregation) for eliciting verbalized confidence from LLMs and benchmarks it on calibration and failure prediction across eight reasoning/knowledge datasets and five LLMs, finding overconfidence and weak failure prediction.
Problem
Black-box LLMs (especially closed-source APIs) lack token likelihoods, so confidence must be elicited from text; it is unclear how well verbalized confidence is calibrated and how well it predicts failures.
Approach
A unified framework decomposing confidence elicitation into prompting strategies (human-inspired prompts such as CoT, Self-Probing, Multi-Step, Top-K), sampling strategies (self-random, paraphrase prompting, misleading hints), and aggregation strategies (consistency, Avg-Conf, Pair-Rank via an MLE over top-K rankings), benchmarked systematically.
Key results
- Vanilla verbalized confidence is overconfident: GPT-3, Vicuna and GPT-3.5 average ECE above 0.377, and GPT-4 averages AUROC of only 62.7%.
- Consistency with M=5 sampled responses on GPT-3.5 raises GSM8K AUROC from 54.8% to 92.7%.
- Pair-Rank aggregation reaches average ECE 0.069 on GPT-4 with Top-K prompting, and Avg-Conf gives the best AUROC on most datasets; white-box token probabilities reach AUROC only about 0.522 to 0.605.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use five self-random responses with consistency aggregation instead of one-response CoT confidence | GPT-3.5 confidence elicitation · GSM8K | AUROC: 54.8 -> 92.7 | Helped |
| Increase the number of self-random responses from M=1 to M=5 and aggregate by consistency (ablation) | GPT-3.5 confidence elicitation · Average across five datasets | AUROC: 56.4 -> 73.0 | Helped |
| Replace consistency-only aggregation with Pair-Rank (ablation) | GPT-4 with Top-K prompt and Self-Random sampling · Mean & Var | ECE: 12.0 -> 6.90 | Helped |
| Use Avg-Conf aggregation instead of consistency-only aggregation (ablation) | GPT-4 with Top-K prompt and Self-Random sampling · Mean & Var | AUROC: 66.9 vs 66.9 | No clear effect |
| Add zero-shot Chain-of-Thought prompting (ablation) | GPT-3.5 verbalized confidence · GSM8K | ECE: 66 -> 10 | Helped |
| Add zero-shot Chain-of-Thought prompting (ablation) | GPT-3.5 verbalized confidence · GSM8K | AUROC: 65 -> 55 | Hurt |
| Use Multi-Step confidence aggregation across reasoning steps (ablation) | GPT-3.5 verbalized confidence · DateUnd | AUROC: 57 -> 72 | Helped |
| Use Multi-Step confidence aggregation across reasoning steps (ablation) | GPT-3.5 verbalized confidence · GSM8K | ECE: 10 -> 22 | Hurt |
| Use Top-K prompt to elicit multiple guesses and confidences (ablation) | GPT-3.5 verbalized confidence · GSM8K | AUROC: 54.8 -> 58.5 | Helped |
| Use weak-claim misleading hints rather than strong-claim hints (ablation) | GPT-3.5 Induced Consistency Confidence · StrategyQA | AUROC: 61.4 -> 62.0 | Helped |
| Use weak-claim misleading hints rather than external-source hints (ablation) | GPT-3.5 verbalized-consistency confidence · StrategyQA | AUROC: 63.4 -> 65.4 | Helped |
| Use key-token probability rather than verbalized confidence | GPT-3 white-box confidence estimation · StrategyQA | AUROC: 50.34 -> 60.61 | Helped |
Surprising findings
- Verbalized confidence values cluster at 80-100% in multiples of 5, mimicking human confidence expressions.
- Low ECE can be achieved by assigning 100% confidence to all samples, which gives no failure discrimination (AUROC near 0.5).
Limitations
- Covers only fixed-form and free-form QA with unique ground-truth answers, not summarization or open-ended generation.
- Focuses on black-box settings; white-box methods perform better though the gap is narrow.
- No single prompting or aggregation strategy consistently wins across datasets and models.
Experimental setup
Datasets: SportUND StrategyQA GSM8K SVAMP DateUnd ObjectCou MMLU Professional Law MMLU Business Ethics
Benchmarks: SportUND StrategyQA GSM8K SVAMP DateUnd ObjectCou MMLU Professional Law MMLU Business Ethics
Metrics: ECE AUROC AUPRC-Positive AUPRC-Negative
Baselines: vanilla verbalized confidence token-probability white-box methods seq-prob len-norm-prob token-prob
Builds on: chain-of-thought top-k prompting self-consistency
Closest papers by meaning
- SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.92)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.91)
- Learning to Route LLMs with Confidence Tokens (ICML 2025 · similarity 0.91)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.91)
- LLMs Show No Signs Of Individuated Metacognition (NeurIPS 2026 · similarity 0.90)
- ConfTuner: Training Large Language Models to Express Their Confidence Verbally (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.