Research map / Reinforcement learning / RL from feedback
Human Alignment of Large Language Models through Online Preference Optimisation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that online IPO's stationary points equal the Nash equilibrium of a regularised preference game (self-play equivalence) and proposes IPO-MD, which samples from a geometric mixture of the online and reference policies, evaluated on TL;DR-style summarisation with T5X models.
Problem
Preference optimisation for LLM alignment: IPO is offline and contrastive while Nash-MD-PG is online and game-theoretic; the relationship between them and how to combine their strengths is unclear.
Approach
Shows that IPO's loss computed on-policy (online IPO) has stationary points at the Nash equilibrium of the regularised preference game and matches self-play in expectation; introduces IPO-MD, which samples data from a geometric mixture of the online and reference policies, interpolating like Nash-MD.
Key results
- In side-by-side summarisation, IPO (0.500 self-match) beats DPO with 0.608 average preference, Nash-MD-PG with 0.621, and RL with 0.791.
- IPO-MD achieves 0.600 vs DPO and 0.608 vs Nash-MD-PG, statistically indistinguishable from IPO once standard deviation is considered.
- Online IPO's expected gradient equals self-play's update direction (Proposition), and its stationary point is the Nash equilibrium of the regularised preference game.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Online IPO rather than IPO-MD | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.515 vs 0.500 | No clear effect |
| Use Online IPO rather than DPO | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.608 | Helped |
| Use Online IPO rather than Nash-MD-PG | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.621 | Helped |
| Use Online IPO rather than SLiC | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.608 | Helped |
| Use Online IPO rather than RL | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.791 | Helped |
| Use IPO-MD rather than DPO | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.600 | Helped |
| Use IPO-MD rather than Nash-MD-PG | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.608 | Helped |
| Use IPO-MD rather than SLiC | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.594 | Helped |
| Use IPO-MD rather than RL | T5X-L policy fine-tuning · Summarisation | preference win rate: 0.778 | Helped |
| Increase the regularisation parameter τ (ablation) | Online IPO · Summarisation | performance against RL | Mixed |
| Increase the regularisation parameter τ (ablation) | IPO versus DPO · Summarisation | side-by-side score | Hurt |
| Use online rather than offline training (ablation) | IPO and DPO · Summarisation | performance against RLHF and SFT | Helped |
Surprising findings
- Offline IPO and Nash-MD-PG, despite being seemingly unrelated, share the same stationary point when IPO is run online.
- Online DPO's stationary points are generally not the regularised Nash equilibrium; no 2-action game satisfies the condition except uniform preferences.
Limitations
- Experiments limited to a 770M-parameter policy on a single summarisation task.
- IPO-MD sampling uses a one-step logit mixture which is not exactly the geometric mixture policy in theory.
Experimental setup
Datasets: TL;DR summarisation preference dataset (92,820 train examples) OpenAI summarisation dataset (SFT) XSum
Benchmarks: XSum validation/test prompts side-by-side summarisation
Metrics: side-by-side win rate standard deviation across 3x3 seed comparisons
Baselines: RL (regularised policy gradient) IPO DPO SLiC Nash-MD-PG
Builds on: IPO Nash-MD-PG DPO SLiC RLHF
Closest papers by meaning
- Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.92)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.91)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.