Research map / Large language models / Alignment and preferences
Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Magnetic Preference Optimization (MPO), an RLHF self-play method built on Magnetic Mirror Descent that aims for last-iterate convergence to the Nash equilibrium of the original (unregularized) general-preference game by periodically resetting the reference (magnet) policy.
Problem
Self-play RLHF with general (non-Bradley-Terry) preferences converges only in average iterate (Mirror Descent) or last iterate only to the NE of a KL-regularized game, so a single LLM does not reach the NE of the original preference game.
Approach
Adapt Magnetic Mirror Descent to RLHF: use a magnet/reference policy that is periodically replaced by the current policy, so iterated regularized NEs converge to the NE of the original constant-sum preference game, with a single extra model kept in memory and a sequential token-level KL and annealed step size.
Key results
- On PKU-SafeRLHF, cost-model score drops from 6.37 (SFT) to -11.38 after three MPO self-play iterations (lower is safer); w/o self-play reaches only -4.18.
- On Open LLM Leaderboard v2, average rises from 33.16 (SFT) to 33.95 after four MPO iterations on Llama-3-8B.
- MPO without self-play underperforms even the first self-play iteration, which the authors take as evidence of overfitting to the opponent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add three rounds of self-play | MPO on Gemma-2B-SFT · PKU-SafeRLHF | cost: 6.37 -> -11.38 | Helped |
| add three rounds of self-play | MPO on Gemma-2B-SFT · HH-Harmless | cost: 11.33 -> -3.86 | Helped |
| omit self-play (ablation) | MPO on Gemma-2B-SFT · PKU-SafeRLHF | cost: -11.38 -> -4.18 | Hurt |
| omit self-play (ablation) | MPO on Gemma-2B-SFT · HH-Harmless | cost: -3.86 -> 6.41 | Hurt |
| add four rounds of self-play | MPO on Llama-3-8B-SFT · Open LLM Leaderboard v2 Average | score: 33.16 -> 33.95 | Helped |
| add four rounds of self-play | MPO on Llama-3-8B-SFT · IFEval | score: 41.63 -> 42.97 | Helped |
| add four rounds of self-play | MPO on Llama-3-8B-SFT · BBH | score: 48.54 -> 51.38 | Helped |
| add four rounds of self-play | MPO on Llama-3-8B-SFT · Math Hard | score: 4.87 -> 5.06 | Mixed |
| add four rounds of self-play | MPO on Llama-3-8B-SFT · MUSR | score: 42.32 -> 40.87 | Hurt |
| periodically update the reference policy instead of keeping it fixed (ablation) | MPO · safety alignment | performance | Helped |
| replace the KL divergence loss with clipping (ablation) | MPO · safety alignment | performance | Hurt |
| use a KL loss instead of directly modifying the reward with KL regularization (ablation) | MPO versus MPO-RT · MixEval-Hard | score | Helped |
Surprising findings
- Fixed-reference or non-self-play variants degrade relative to self-play, suggesting RLHF with general preferences overfits to the opponent rather than the reward model.
- Gains on Llama-3-8B general capability are modest (average 33.16 to 33.95 on Open LLM Leaderboard v2), with Math Hard not consistently improving.
Limitations
- Theoretical guarantees rely on assumptions (monotone, L-smooth, bounded policy space) and approximate solves of regularized games; practical reference updates use a fixed interval rather than detecting convergence.
- Gains on general capability benchmarks are modest and uneven (e.g., Math Hard and BoolQ fluctuate or decline).
Experimental setup
Datasets: Alpaca PKU-SafeRLHF Anthropic HH-Harmless preference_datase_mixture2_and_safe_pku preference_700K SFT-OpenHermes-2.5-Standard UltraFeedback HelpSteer OpenOrca UltraInteract Capybara prompt-collection-v0.1
Benchmarks: PKU-SafeRLHF cost model HH-Harmless cost model GPT-4o safety win rate MixEval-Hard Open LLM Leaderboard v2 RewardBench
Metrics: cost model score win rate overall score average accuracy
Baselines: SPO NLHF PPO Iterative DPO MPO-RT MPO without self-play
Builds on: Magnetic Mirror Descent Mirror Descent Policy Optimization Nash Learning from Human Feedback Self-play Preference Optimization ReMax
Closest papers by meaning
- Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.94)
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.90)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.89)
- AMPO: Active Multi Preference Optimization for Self-play Preference Selection (ICML 2025 · similarity 0.89)
- Multi-turn Reinforcement Learning with Preference Human Feedback (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.