Research map / Reinforcement learning / RL from feedback
Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INPO, an online no-regret (OMD/Hedge-style) self-play method that learns the Nash policy of a KL-regularized general-preference game via a population least-squares loss over on-policy preference pairs, avoiding win-rate estimation.
Problem
RLHF under general (non-Bradley-Terry) preferences: learning the Nash policy of the alignment game online without estimating expected win rates per response, which are costly to compute or annotate.
Approach
Plays the policy against itself with online mirror descent (Hedge-style KL-regularized update) and converts the closed-form update into a least-squares loss on log-ratio differences over binary preference pairs sampled from the current policy, so the minimizer is the next iterate without computing win rates; each iteration needs only O(K) tournament-based preference queries.
Key results
- INPO (PM) reaches 42.6% length-controlled win rate on AlpacaEval 2.0 and 37.8% on Arena-Hard v0.1 from the LLaMA-3-8B SFT model, beating iterative DPO (PM) at 28.5 / 29.6 and SPPO (PM) at 32.8 / 29.2.
- On academic benchmarks, INPO averages 60.0 vs 59.5 for SPPO, 58.8 for iterative DPO, and 55.5 for the SFT model.
- Removing the KL term (INPO w/o KL) lowers AlpacaEval 2.0 from 42.6 to 41.6 with the preference model oracle, and from 37.6 to 35.4 with the BT reward model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use INPO with the preference model instead of SPPO with the preference model | LLaMA-3-8B SFT alignment · AlpacaEval 2.0 | length-controlled win rate: 32.8 -> 42.6 | Helped |
| use INPO with the preference model instead of SPPO with the preference model | LLaMA-3-8B SFT alignment · Arena-Hard v0.1 | win rate: 29.2 -> 37.8 | Helped |
| use INPO with the preference model instead of SPPO with the preference model | LLaMA-3-8B SFT alignment · MT-Bench | score: 8.26 -> 8.43 | Helped |
| use INPO with the preference model instead of SPPO | LLaMA-3-8B SFT alignment · AVG (six academic benchmarks) | average score: 59.5 -> 60.0 | Helped |
| use a preference model instead of a BT reward model as the preference oracle | INPO · AlpacaEval 2.0 | length-controlled win rate: 37.6 -> 42.6 | Helped |
| use a preference model instead of a BT reward model as the preference oracle | INPO · Arena-Hard v0.1 | win rate: 34.7 -> 37.8 | Helped |
| use a preference model instead of a BT reward model as the preference oracle | INPO · MT-Bench | score: 8.27 -> 8.43 | Helped |
| add KL regularization (ablation) | INPO with the BT reward model oracle · AlpacaEval 2.0 | length-controlled win rate: 35.4 -> 37.6 | Helped |
| add KL regularization (ablation) | INPO with the BT reward model oracle · Arena-Hard v0.1 | win rate: 33.6 -> 34.7 | Helped |
| add KL regularization (ablation) | INPO with the BT reward model oracle · MT-Bench | score: 8.10 -> 8.27 | Helped |
| add KL regularization (ablation) | INPO with the preference model oracle · AlpacaEval 2.0 | length-controlled win rate: 41.6 -> 42.6 | Helped |
| add KL regularization (ablation) | INPO with the preference model oracle · Arena-Hard v0.1 | win rate: 36.5 -> 37.8 | Helped |
Surprising findings
- Preference-model oracles consistently outperform BT reward-model oracles, supporting the value of non-BT general preferences.
- Gains are smaller when starting from the already RLHF-tuned Llama-3-8B-Instruct than from the SFT model.
Limitations
- Theory relies on a bounded log density ratio assumption and is restricted to the non-contextual analysis extended straightforwardly to contexts.
- Experiments use AI preference oracles (reward and preference models) rather than human labels, and GPT-4-based judges for evaluation.
- Only three iterations and a single 8B model family are evaluated; the authors note smaller gains on the instruct model.
Experimental setup
Datasets: RLHFlow iterative-prompt-v1 (20K prompts per iteration, 3 iterations)
Benchmarks: AlpacaEval 2.0 Arena-Hard v0.1 MT-Bench IFEval GPQA MMLU Hellaswag TruthfulQA GSM8K
Metrics: length-controlled win rate win rate MT-Bench score accuracy
Baselines: Iterative DPO (RM) Iterative DPO (PM) SPPO (PM) LLaMA-3-70B-it Mixtral-8x22B-it GPT-4-0613 Claude-3-Opus GPT-4 Turbo
Builds on: DPO Nash-MD SPPO iterative DPO online mirror descent
Closest papers by meaning
- Nash Learning from Human Feedback (ICML 2024 · similarity 0.93)
- Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.92)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- Multi-turn Reinforcement Learning with Preference Human Feedback (NeurIPS 2024 · similarity 0.91)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.90)
- Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.