Research map / Large language models / Language, safety and interpretability

Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DACA, an unsupervised post-hoc temperature-scaling method that calibrates overconfident post-trained LLMs by aligning their confidence to a pre-trained LLM's on unlabeled data, using only examples where the two models agree.

Problem

Post-trained LLMs are overconfident, and post-hoc calibration such as temperature scaling needs labeled data that is scarce for downstream tasks. The paper asks how to calibrate PoLMs using only unlabeled data.

Approach

Align the PoLM's temperature-scaled distribution to the well-calibrated PLM's distribution via KL divergence, but mask out disagreement examples (where PLM and PoLM argmax differ). The theory argues disagreement examples push the temperature upward and cause under-confidence, so excluding them yields a more reasonable temperature.

Key results

• On MMLU averaged over 57 subjects, DACA reduces Gemma-3-12B-Instruct ECE from 23.679% to 8.596%, versus 9.746% for labeled temperature scaling.
• Using Gemma-3-12B as the pre-trained calibrator, DACA reduces GPT-4o ECE on MedMCQA from 21.231% to 6.993%.
• On Llama-3.1-8B post-trained with SFT+DPO, DACA reduces MedMCQA ECE from 25.120% to 5.418%.

Measured effects reported in the paper

Surprising findings

• Naive confidence alignment without disagreement filtering makes the PoLM under-confident, with the temperature increasing toward infinity on disagreement examples.
• Verbalized confidence methods (Elicitation, Elicitation-Ensemble) often perform worse than the vanilla model on ECE.

Limitations

• Requires an extra inference pass through the pre-trained model, adding modest computational cost.
• Filtering disagreement examples reduces the pool of unlabeled calibration data.
• The Proposition on temperature tending to infinity relies on simplifying assumptions and a loosely argued proof.

Experimental setup

Datasets: MMLU MedMCQA MathQA TruthfulQA

Benchmarks: MMLU MedMCQA MathQA TruthfulQA

Metrics: ECE MCE AECE Brier Score

Baselines: CAPE Elicitation Elicitation-Ensemble Temperature Scaling Vanilla

Builds on: temperature scaling vector scaling matrix scaling Platt scaling P(True)

Closest papers by meaning

• Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.91)
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.90)
• SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.90)
• Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.89)
• Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.89)
• Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.89)