Research map / Large language models / Alignment and preferences

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a unified theoretical view that SFT is a special case of implicit reward learning shared with DPO, and uses this to motivate smaller SFT learning rates and alternative f-divergence SFT objectives that improve post-DPO instruction-following performance.

Problem

The theoretical relationship between SFT and preference learning (DPO) in LLM post-training is poorly understood, and SFT is usually treated as a warm-up step; the KL term in SFT derived from distribution matching appears constant w.r.t. the policy.

Approach

Derives via non-adversarial distribution matching with KL regularization that SFT (total-variation case) optimizes the same optimal policy-reward subspace as DPO, argues the KL term is zero-order and so does not constrain updates, reduces the SFT learning rate to compensate, and proposes Pearson chi-square and squared Hellinger SFT objectives that retain the KL term.

Key results

• Reducing SFT learning rate from 2e-5 to 1e-6 for Mistral raised SimPO AlpacaEval 2 win rate from 22.9% to 24.0% (LC 24.1 to 27.3) and DPO WR from 12.5% to 16.7% with Arena-Hard gain of ~9.9 points.
• For Llama-3-8B, smaller-lr SFT followed by DPO improved AlpacaEval 2 WR from 14.2% to 16.2% and SimPO WR from 21.3% to 25.0%.
• Pearson-chi2 SFT followed by DPO gave Llama-3 Arena-Hard 24.7 versus 21.1 for standard SFT+DPO, showing weaker SFT can yield better post-DPO results.

Measured effects reported in the paper

Surprising findings

• A better SFT checkpoint does not necessarily lead to better DPO results; weaker SFT with alternative f-divergences yields stronger post-DPO models.
• Interleaved SFT and DPO training caused a significant performance drop after each DPO stage, indicating conflicting objectives in their setting.

Limitations

• Alternative divergences (JS, KL, reverse KL) were not experimentally validated due to numerical instability.
• Experiments limited to instruction following on two 7-8B base models with AlpacaEval/Arena-Hard/MT-Bench; reliance on approximations such as the value-dominance assumption.
• Multi-objective SFT+DPO joint training attempts failed and are only reported in the appendix.

Experimental setup

Datasets: UltraChat-200K UltraFeedback MATH-500

Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench MATH-500

Metrics: length-controlled win rate win rate MT-Bench score Kendall rank correlation

Baselines: SFT DPO SimPO RRHF SLiC-HF CPO IPO KTO ORPO R-DPO

Builds on: Direct Preference Optimization SimPO non-adversarial imitation learning maximum-entropy RL

Closest papers by meaning

• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.92)
• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.91)
• Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.90)
• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (NeurIPS 2024 · similarity 0.89)
• Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models (ICML 2025 · similarity 0.89)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.89)