Research map / Large language models / Alignment and preferences

Self-Play Preference Optimization for Language Model Alignment

ICLR 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-Play Preference Optimization (SPPO), an iterative self-play method that approximates the Nash equilibrium of a constant-sum preference game using a squared-loss update, fine-tuning Mistral-7B and Llama-3-8B with a PairRM preference model.

Problem

RLHF with Bradley-Terry reward models cannot capture intransitive human preferences, and existing general-preference methods (DPO/IPO variants) lack provable convergence to the Nash equilibrium or scale poorly; the paper targets efficient LLM alignment under general preference.

Approach

Derive an exponential-weights multiplicative update for a two-player constant-sum game over responses and approximate it with a square loss regressing the log-ratio of the policy to the win rate against the previous policy, replacing the log-partition term with a constant eta/2; this yields a policy-gradient-like, semi-online self-play update.

Key results

• Mistral-7B-SPPO Iter3 reaches a 28.53% length-controlled win rate against GPT-4-Turbo on AlpacaEval 2.0 (31.02% raw), above Snorkel's 26.39% LC.
• Llama-3-8B-SPPO Iter3 reaches a 38.77% LC win rate on AlpacaEval 2.0 versus 22.92% for the base Llama-3-8B-Instruct.
• Mistral-7B-SPPO Iter3 achieves an average Open LLM Leaderboard score of 66.75 at Iter2 (state of the art), and Llama-3-8B-SPPO Iter3 reaches 70.29.

Measured effects reported in the paper

Surprising findings

• MT-Bench performance drops in the first two SPPO iterations before recovering at Iter3, which the authors could not explain.
• Open LLM Leaderboard scores for Mistral-7B DPO, IPO, and SPPO decline after the first or second iteration, suggesting alignment with PairRM may hurt general capabilities.

Limitations

• Theory assumes the model class is expressive enough and generated data covers the input space.
• Experiments use a single dataset (UltraFeedback) and a limited set of benchmarks due to compute constraints.
• Gains on general capability benchmarks (Open LLM Leaderboard) do not persist across iterations.

Experimental setup

Datasets: UltraFeedback

Benchmarks: AlpacaEval 2.0 Arena-Hard MT-Bench Open LLM Leaderboard

Metrics: length-controlled win rate win rate average score accuracy

Baselines: Iterative DPO Iterative IPO Snorkel (Mistral-PairRM-DPO) Self-Rewarding LM Mistral-7B-Instruct-v0.2 Llama-3-8B-Instruct

Builds on: DPO IPO Nash-MD Self-Rewarding Language Models PairRM

Closest papers by meaning

• Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment (ICLR 2025 · similarity 0.94)
• Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.92)
• AMPO: Active Multi Preference Optimization for Self-play Preference Selection (ICML 2025 · similarity 0.92)
• Human Alignment of Large Language Models through Online Preference Optimisation (ICML 2024 · similarity 0.92)
• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
• Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)