Research map / Large language models / Alignment and preferences
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cal-DPO, which adds a squared-loss calibration term to DPO (and IPO/SLiC) so that implicit rewards match the scale of ground-truth rewards, improving alignment and keeping chosen-response likelihood from decreasing.
Problem
Contrastive preference optimization (DPO, IPO, SLiC) only optimizes relative differences between implicit rewards of chosen and rejected responses, ignoring absolute reward values, so the likelihood of chosen responses can keep decreasing and hurt alignment, especially on reasoning tasks.
Approach
Add a squared-loss calibration term that regresses each implicit reward (log policy ratio) toward ground-truth reward divided by beta (using +1/2beta for chosen and -1/2beta for rejected when only pairwise labels exist), making the implicit rewards comparable in scale to true rewards; implementable in one line on top of DPO.
Key results
- On UltraFeedback with Zephyr-7b-sft, Cal-DPO reaches 21.72 on IFEval vs 10.49 for DPO and 2.42 on MATH vs 1.36 for DPO, with best scores across all listed reasoning benchmarks.
- On TL;DR summarization, Cal-DPO achieves 75.61% win rate vs SFT and 59.37% vs chosen against GPT-4 judging, versus 71.22% and 57.58% for DPO.
- On IMDb, Cal-DPO attains reward 0.645 and perplexity 32.31, better than PPO (0.626, 35.05) and DPO (0.617, 34.21).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reward calibration to DPO | DPO with Zephyr-7B-SFT · IFEval | score: 13.32 -> 21.72 | Helped |
| Add reward calibration to DPO | DPO with Zephyr-7B-SFT · MATH | score: 2.15 -> 2.42 | Helped |
| Add reward calibration to DPO | DPO with Zephyr-7B-SFT · GSM8K | score: 33.74 -> 34.87 | Helped |
| Add reward calibration to DPO | DPO fine-tuning · TL;DR summarization | average win rate: 66.01 -> 67.49 | Helped |
| Add reward calibration to DPO | DPO fine-tuning · Anthropic-HH | average win rate: 67.86 -> 69.07 | Helped |
| Add reward calibration to DPO | DPO fine-tuning · IMDb sentiment | reward: 0.632 -> 0.645 | Helped |
| Add reward calibration to DPO | DPO fine-tuning · IMDb sentiment | perplexity: 34.08 -> 32.31 | Helped |
| Calibrate the implicit reward when using IPO | IPO · Anthropic-HH | performance | Helped |
| Calibrate the implicit reward when using SLiC | SLiC · Anthropic-HH | performance | Helped |
| Increase β too much (ablation) | Cal-DPO · Various tasks | model performance | Hurt |
| Use a small β rather than a larger β (ablation) | Cal-DPO · Various tasks | model performance | Helped |
Surprising findings
- In DPO training, chosen-response rewards fall below zero while Cal-DPO's keep increasing and stay positive, indicating chosen-response likelihood decline is a systematic issue.
- Cal-DPO outperforms CPO, which adds an NLL term on chosen responses, suggesting calibrating both chosen and rejected rewards matters more than only raising chosen likelihood.
Limitations
- Limited to offline methods; does not consider on-policy learning with a reward model during training.
- Pairwise-only setting uses heuristic pseudo-rewards of +1/2 and -1/2 rather than true ground-truth rewards.
- Performance is sensitive to beta; too large beta keeps policy close to reference and degrades performance.
Experimental setup
Datasets: UltraFeedback Binarized Reddit TL;DR summarization Anthropic-HH IMDb
Benchmarks: MMLU-PRO IFEval BBH GPQA MATH GSM8K ARC AlpacaEval 2.0 Reddit TL;DR win rate Anthropic-HH win rate IMDb reward and perplexity
Metrics: accuracy win rate length-controlled win rate reward perplexity
Baselines: DPO IPO SLiC CPO f-DPO DPOP DPO+NLL PPO SFT
Builds on: DPO IPO SLiC RLHF Bradley-Terry model
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.94)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.94)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.94)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.