Research map / Large language models / Language, safety and interpretability
Confidence is Not Universal: Task-Dependent Calibration and Emergent Behavior in LLMs
Chaeyun Jang, Moonseok Choi, Yegon Kim, Seungyoo Lee, Juho Lee, Hyungi Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| supervise confidence on GSM8K and test on retrieval (RULER) tasks | CSFT, Llama-3.2-3B-Instruct · Cluster 2 average | ECE: 0.2745 -> 0.6375 | Hurt |
| supervise confidence on RULER-4k and test on reasoning tasks | CSFT, Llama-3.2-3B-Instruct · Cluster 1 average | ECE: 0.3390 -> 0.4288 | Hurt |
| supervise confidence on GSM8K and test within reasoning family | CSFT, Llama-3.2-3B-Instruct · Cluster 1 average | ECE: 0.3390 -> 0.1086 | Improved |
| disentangle confidence into reasoning and evidence confidence, trained on both clusters | CSFT, Llama-3.2-3B-Instruct · RULER-8k | ECE: 0.6351 -> 0.1466 vs CSFT on GSM8K | Improved |
| disentangle confidence, evaluated on held-out hybrid ContractNLI | CSFT, Llama-3.2-3B-Instruct · ContractNLI | ECE: 0.5021 -> 0.1181 | Improved |
| use full-parameter fine-tuning instead of LoRA (ablation) | CSFT GSM8K to RULER-4K · RULER-4K | ECE: ECE 0.60, AUROC near chance | No clear effect |
| add CSFT on GSM8K | Llama-3.2-3B-Instruct · GSM8K evaluation | accuracy: 69% -> 75% | Improved |
| place confidence prompt before reasoning (prefix) instead of after the answer (suffix) (ablation) | CSFT on GSM8K · GSM8K | accuracy | Hurt |
| remove KL regularization with suffix confidence prompt (ablation) | CSFT on GSM8K · GSM8K | accuracy | No clear effect |
| randomize confidence labels (remove correct labels) (ablation) | CSFT on GSM8K · GSM8K | accuracy: -2.14 ACC, +0.05 ECE | Hurt |
| add explicit confidence question prompt (ablation) | CSFT on GSM8K · GSM8K | training stability: training collapsed without it | Improved |
| use adaptive labels instead of fixed labels (ablation) | CSFT on HellaSwag · HellaSwag ID, MATH, MMLU | AUROC, ECE, Brier: ID AUROC -0.09; MATH AUROC +0.01; MMLU AUROC +0.06 | No clear effect |
Closest papers by meaning
- Reasoning Models Better Express Their Confidence (NeurIPS 2025 · similarity 0.91)
- LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning (NeurIPS 2026 · similarity 0.90)
- RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models (ICLR 2026 · similarity 0.89)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.89)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.88)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
