Research map / Large language models / Alignment and preferences
Nash Learning from Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Nash learning from human feedback (NLHF), learning a pairwise preference model and computing its regularized Nash equilibrium with Nash-MD (tabular, last-iterate convergence) and policy-gradient variants, evaluated on TL;DR summarization.
Problem
Reward-model-based RLHF cannot represent non-transitive or diverse human preferences and its reward depends on the data distribution, so aligning LLMs with pairwise human feedback needs a different objective.
Approach
Replace the Bradley-Terry reward model with a pairwise preference model and optimize for its regularized Nash equilibrium; Nash-MD does mirror descent against a geometric mixture of the current and reference policy, giving last-iterate convergence without storing past policies.
Key results
- Nash-MD-PG with mixture parameter beta in [0.125, 0.375] outperforms Self-Play and Best-Response in PaLM 2 pairwise preference table.
- Nash-MD tabular algorithm has last-iterate KL convergence at rate O(1/T) to the regularized Nash equilibrium.
- Table shows MD1 (beta=0.125) beats SFT with preference 0.982 and RLHF with 0.598.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Nash-MD-PG with β=0.125 instead of RLHF | SFT-initialized summarization policy · TL;DR | PaLM 2 preference win rate: 0.598 | Helped |
| Use Nash-MD-PG with β=0.125 instead of Self-Play | SFT-initialized summarization policy · TL;DR | PaLM 2 preference win rate: 0.592 | Helped |
| Use Nash-MD-PG with β=0.25 instead of Self-Play (ablation) | Nash-MD-PG · TL;DR | PaLM 2 preference win rate: 0.504 | No clear effect |
| Use Nash-MD-PG with β=0.375 instead of Self-Play (ablation) | Nash-MD-PG · TL;DR | PaLM 2 preference win rate: 0.545 | Helped |
| Use Nash-MD-PG with β=0.5 instead of Self-Play (ablation) | Nash-MD-PG · TL;DR | PaLM 2 preference win rate: 0.499 | No clear effect |
| Use Nash-MD-PG with β=0.625 instead of Self-Play (ablation) | Nash-MD-PG · TL;DR | PaLM 2 preference win rate: 0.451 | Hurt |
| Use Nash-MD-PG with β=0.75 instead of Self-Play (ablation) | Nash-MD-PG · TL;DR | PaLM 2 preference win rate: 0.310 | Hurt |
| Use best response against SFT (β=1) instead of Self-Play (ablation) | Nash-MD-PG · TL;DR | PaLM 2 preference win rate: 0.211 | Hurt |
| Use Nash-EMA-PG last iterate with β=0.999 instead of Self-Play (ablation) | Nash-EMA-PG · TL;DR | PaLM 2 preference win rate: 0.445 | Hurt |
| Use Nash-EMA-PG last iterate with β=0.9995 instead of Self-Play (ablation) | Nash-EMA-PG · TL;DR | PaLM 2 preference win rate: 0.362 | Hurt |
| Use average Nash-EMA-PG policy with β=0.999 instead of Self-Play (ablation) | Nash-EMA-PG · TL;DR | PaLM 2 preference win rate: 0.464 | Hurt |
| Use average Nash-EMA-PG policy with β=0.9995 instead of Self-Play (ablation) | Nash-EMA-PG · TL;DR | PaLM 2 preference win rate: 0.488 | No clear effect |
Surprising findings
- Self-Play and Best-Response against a fixed policy are both outperformed by intermediate mixture parameters.
Limitations
- Hyperparameters were not tuned per method, so comparisons with RLHF are not fully fair.
- Experiments limited to one summarization task with PaLM 2 Large as judge.
- One-step-at-a-time geometric mixture sampling differs from the true regularized policy.
Experimental setup
Datasets: TL;DR
Benchmarks: TL;DR summarization pairwise evaluation
Metrics: pairwise win probability vs PaLM 2 Large preference
Baselines: RLHF Self-Play Best-Response against SFT Nash-EMA
Builds on: RLHF Direct Preference Optimization Identity Preference Optimization online mirror descent fictitious play
Closest papers by meaning
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.93)
- Multi-turn Reinforcement Learning with Preference Human Feedback (NeurIPS 2024 · similarity 0.92)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.91)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.