AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Confidence is Not Universal: Task-Dependent Calibration and Emergent Behavior in LLMs

Chaeyun Jang, Moonseok Choi, Yegon Kim, Seungyoo Lee, Juho Lee, Hyungi Lee

ICML 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
supervise confidence on GSM8K and test on retrieval (RULER) tasksCSFT, Llama-3.2-3B-Instruct · Cluster 2 averageECE: 0.2745 -> 0.6375Hurt
supervise confidence on RULER-4k and test on reasoning tasksCSFT, Llama-3.2-3B-Instruct · Cluster 1 averageECE: 0.3390 -> 0.4288Hurt
supervise confidence on GSM8K and test within reasoning familyCSFT, Llama-3.2-3B-Instruct · Cluster 1 averageECE: 0.3390 -> 0.1086Improved
disentangle confidence into reasoning and evidence confidence, trained on both clustersCSFT, Llama-3.2-3B-Instruct · RULER-8kECE: 0.6351 -> 0.1466 vs CSFT on GSM8KImproved
disentangle confidence, evaluated on held-out hybrid ContractNLICSFT, Llama-3.2-3B-Instruct · ContractNLIECE: 0.5021 -> 0.1181Improved
use full-parameter fine-tuning instead of LoRA (ablation)CSFT GSM8K to RULER-4K · RULER-4KECE: ECE 0.60, AUROC near chanceNo clear effect
add CSFT on GSM8KLlama-3.2-3B-Instruct · GSM8K evaluationaccuracy: 69% -> 75%Improved
place confidence prompt before reasoning (prefix) instead of after the answer (suffix) (ablation)CSFT on GSM8K · GSM8KaccuracyHurt
remove KL regularization with suffix confidence prompt (ablation)CSFT on GSM8K · GSM8KaccuracyNo clear effect
randomize confidence labels (remove correct labels) (ablation)CSFT on GSM8K · GSM8Kaccuracy: -2.14 ACC, +0.05 ECEHurt
add explicit confidence question prompt (ablation)CSFT on GSM8K · GSM8Ktraining stability: training collapsed without itImproved
use adaptive labels instead of fixed labels (ablation)CSFT on HellaSwag · HellaSwag ID, MATH, MMLUAUROC, ECE, Brier: ID AUROC -0.09; MATH AUROC +0.01; MMLU AUROC +0.06No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy