Research map / Large language models / Instruction and fine-tuning
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that preference-aligned LLMs become overconfident on multiple-choice questions and proposes calibration-aware fine-tuning (CFT) and regularized CFT with an EM-based ECE term to restore calibration while keeping accuracy.
Problem
Preference alignment (RLHF/DPO) makes LLMs poorly calibrated, with overconfident predictions on multiple-choice questions, even though pre-trained models are well calibrated.
Approach
Frames calibration as a constrained accuracy-maximization problem with ECE=0, introduces calibratable vs non-calibratable regimes, and fine-tunes aligned models with domain-specific QA SFT loss plus an EM-estimated ECE regularizer (RCFT) to keep calibration as accuracy rises.
Key results
- Reduces in-domain conf-ECE of Llama3.1-8B-Tulu from 0.1953 (DPO) to 0.0239 with CFT.
- Across the four models, the proposed methods reduce preference-aligned ECE from 14.22%-20.10% to 2.39%-6.51% per the authors' summary.
- RCFT raises in-domain accuracy on Llama3.1-8B-Tulu from 0.6228 (DPO) to 0.8341 at the cost of higher conf-ECE (0.0897).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with SFT₁ using domain-specific question-answering data | Preference-aligned LLMs · In-domain, four evaluated models | conf-ECE: 0.1422 -> 0.0379; 0.201 -> 0.0651 | Helped |
| Fine-tune with SFT₁ using domain-specific question-answering data | Olmo2-7B · Arc-Challenge (out-domain) | conf-ECE: 0.1160 -> 0.0225 | Helped |
| Fine-tune with SFT₂ and ECE regularization | Preference-aligned LLMs · In-domain | accuracy: 0.6228 -> 0.8341; 0.6210 -> 0.8510 | Helped |
| Use SFT₂ without ECE regularization (ablation) | RCFT · Fine-tuning evaluation | conf-ECE: null | Hurt |
| Use only ECE loss, omitting the SFT accuracy objective (ablation) | Calibration-aware fine-tuning · Fine-tuning evaluation | accuracy: null | Hurt |
Surprising findings
- Using only the ECE loss drives the model toward random guessing (near-zero ECE, 25% accuracy).
- Temperature scaling sometimes gives lower out-domain conf-ECE than CFT (e.g., Llama3.1 out-domain 0.0679 vs CFT 0.0688 and Vicuna 0.0224 vs 0.0331).
Limitations
- Study limited to multiple-choice settings; free-form generation not addressed.
- Theoretical constant C in the TCE lower bound is undetermined; regime analysis relies on assumptions (e.g., policy can represent arbitrary distributions).
- Not evaluated on poorly calibrated initial models or quantized/black-box/LoRA-specific settings.
Experimental setup
Datasets: MMLU MedMCQA OpenBookQA ARC-Challenge AlpacaEval Arena-Hard UltraFeedback-Binarized-Preferences
Benchmarks: MMLU MedMCQA OpenBookQA ARC-Challenge AlpacaEval Arena-Hard UltraFeedback
Metrics: conf-ECE cw-ECE accuracy win rate
Baselines: DPO RLHF Temperature Scaling Label Smoothing
Builds on: DPO RLHF temperature scaling QLoRA
Closest papers by meaning
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.92)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.91)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.91)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.91)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.