Research map / Reinforcement learning / RL from feedback
A Minimaximalist Approach to Reinforcement Learning from Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPO, a reward-model-free RLHF method that reduces computing a symmetric Minimax Winner to running one no-regret learner against its own iterates, implemented in practice as win-rate reward relabeling against a queue of past trajectories.
Problem
Standard reward-based RLHF assumes a total order over behavior, so it cannot represent intransitive, non-Markovian or stochastic preferences and collapses diversity; offline preference methods like DPO also suffer compounding errors and adversarial dueling methods are unstable.
Approach
Frame preference learning as a two-player zero-sum game whose Minimax Winner can be computed by a single agent playing against itself, exploiting anti-symmetry of the preference function; the per-trajectory reward is the win rate against on-policy samples (a policy-dependent reward), used with off-the-shelf deep RL (SAC/PPO).
Key results
- On discrete intransitive preferences, SPO almost exactly recovers the closed-form Minimax Winner across sub-population weightings, whereas reward modeling converges to a corner of the simplex.
- On DMControl with ground-truth-reward preferences and Ant-v3 non-Markovian constraints, SPO learns more sample-efficiently than reward-model baselines and more consistently crosses the 4·r_max threshold.
- With Bernoulli preference noise, SPO performs comparably to RM on some environments but noticeably worse on others.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPO instead of iterative reward modeling | Preference optimization on discrete intransitive preferences · Discrete intransitive preferences | Minimax Winner accuracy | Helped |
| Optimize using self-play preference win rates | SPO · Ant-v3 intransitive preferences | Qualitative match to Minimax Winner | Helped |
| Use SPO instead of iterative reward modeling | Maximum-reward preference learning · DMControl maximum-reward preferences | Sample efficiency | Mixed |
| Use SPO instead of iterative reward modeling | Noisy preference learning · DMControl noisy preferences | Performance | Mixed |
| Use SPO instead of iterative reward modeling | Non-Markovian preference learning · Non-Markovian preferences | Threshold attainment | Helped |
| Freeze the reward model partway through training (ablation) | Iterative RM · Continuous control preference learning | Performance | Hurt |
| Use snippet-level feedback instead of trajectory-level comparisons (ablation) | Reward-model baseline · Continuous control | Performance | Mixed |
Surprising findings
- Under Bernoulli label noise, SPO is less robust than the reward-model baseline on some environments, since it lacks the averaging effect of a learned model.
- The paper constructs a preference matrix where DPO provably cannot recover the Minimax Winner for any beta, even with a uniform reference policy.
Limitations
- Experiments are limited to context-free continuous control and discrete toy problems; the contextual LLM setting is not empirically explored.
- Last-iterate convergence under bandit feedback and uncertainty in learned preference functions are left open.
- Performs worse than reward modeling under noisy preferences in some environments.
Experimental setup
Datasets: DMControl Mujoco Ant-v3
Benchmarks: DMControl Mujoco Ant-v3 discrete three-option intransitive preference problem
Metrics: distance to exact minimax winner sample efficiency / return robustness to Bernoulli preference noise
Baselines: Reward modeling (RM) with iterative updates DPO-style reward-free methods (analyzed theoretically)
Builds on: DPO Soft Actor-Critic PPO Hedge / no-regret learning Bradley-Terry reward modeling
Closest papers by meaning
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.91)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.91)
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.91)
- Online Iterative Reinforcement Learning from Human Feedback with General Preference Model (NeurIPS 2024 · similarity 0.91)
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.