Research map / Large language models / Alignment and preferences

SimPO: Simple Preference Optimization with a Reference-Free Reward

NeurIPS 2024 · Large language models › Alignment and preferences · approach: DPO design choices · 39 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SimPO, a reference-free preference optimization objective that uses length-normalized average log-likelihood as the implicit reward plus a target reward margin in the Bradley-Terry loss, outperforming DPO and other offline methods on AlpacaEval 2, Arena-Hard, and MT-Bench.

Problem

Offline preference optimization such as DPO uses a reference-model-based implicit reward that mismatches the average log-likelihood used at generation time, and requires an extra reference model forward pass, adding compute and memory cost.

Approach

Replace DPO's reference-ratio reward with the length-normalized average log-likelihood of the policy (aligned with generation), and add a target reward margin gamma to the Bradley-Terry objective so the winning response must exceed the losing one by a margin.

Key results

• SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and up to 7.5 points on Arena-Hard across four settings.
• Gemma-2-9B-it-SimPO reaches 72.4% length-controlled win rate on AlpacaEval 2 and 59.1% on Arena-Hard, ranking 1st among <10B models on Chatbot Arena.
• Removing length normalization drops Mistral-Base AlpacaEval 2 LC win rate from 21.5% to 11.9%; SimPO cuts run time ~20% and GPU memory ~10% versus DPO on 8xH100.

Measured effects reported in the paper

Surprising findings

• Only about 50% of DPO-trained training triples satisfy the likelihood ranking implied by the DPO reward, indicating random ranking accuracy in average log-likelihood.
• Many DPO variants did not empirically outperform standard DPO when hyperparameters were thoroughly tuned.
• Preference optimization generally decreased GSM8K math performance, with SimPO sometimes comparable or worse than DPO.

Limitations

• Lacks rigorous theoretical analysis; the target reward margin is an extra hyperparameter requiring manual tuning.
• Does not explicitly address safety and honesty; relies on UltraFeedback which is mainly helpfulness-focused.
• Underperforms DPO on some reasoning tasks such as GSM8K, possibly due to lack of SFT regularization.

Experimental setup

Datasets: UltraChat-200k UltraFeedback Open LLM Leaderboard tasks

Benchmarks: AlpacaEval 2 Arena-Hard v0.1 MT-Bench Open LLM Leaderboard (MMLU, ARC, HellaSwag, TruthfulQA, Winograd, GSM8K) Chatbot Arena

Metrics: length-controlled win rate raw win rate win rate MT-Bench score reward accuracy Spearman correlation accuracy

Baselines: DPO IPO CPO KTO ORPO R-DPO RRHF SLiC-HF

Builds on: DPO Zephyr training pipeline IPO ORPO Bradley-Terry model

Closest papers by meaning

• MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge (NeurIPS 2026 · similarity 0.91)
• MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.91)
• TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization (ICML 2025 · similarity 0.90)
• Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.90)
• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.90)
• DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.90)