Research map / Reinforcement learning / RL from feedback
Learning Optimal Advantage from Preferences and Mistaking It for Reward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that learning from regret-based human preferences while mistakenly assuming a partial-return preference model yields an approximation of the optimal advantage function that acts as a highly shaped reward, and that greedy maximization of the learned advantage is the more appropriate approach.
Problem
Reward learning from pairwise trajectory-segment preferences assumes partial-return preference models, but humans may actually give preferences based on regret; the consequences of this mismatch (what is learned and how it is used) are unclear, and this mismatch underlies RLHF fine-tuning of LLMs.
Approach
Reframes partial-return-based reward learning as learning an approximation of the optimal advantage function when preferences are regret-based; analyzes using this learned advantage as a reward (policy invariance under zero-max shift, reward shaping, termination bias from loops) and recommends greedy maximization of the learned advantage; reinterprets RLHF LLM fine-tuning (ChatGPT, Sparrow, Llama
Key results
- Shifting the learned advantage so max over actions is zero yields no difference between greedy advantage and reward-based policies across 90 gridworld MDPs.
- Including transitions from absorbing states significantly improves greedy A*_r performance for most training set sizes (p < 0.0007).
- Q-learning with the learned advantage as reward is more sample efficient than with the ground-truth reward (Wilcoxon p < 0.00003).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| shift the learned optimal advantage so its maximum action value is 0 in every state (ablation) | greedy Ahat and Ahat-as-reward · 90 small gridworld MDPs | normalized mean return | No clear effect |
| include transitions from absorbing states in preference segments (ablation) | greedy Ahat · 30 randomly generated gridworld MDPs; noiseless preferences | normalized mean return | Helped |
| include transitions from absorbing states in preference segments (ablation) | Ahat-as-reward · 30 randomly generated gridworld MDPs; noiseless preferences | normalized mean return | No clear effect |
| include transitions from absorbing states in preference segments (ablation) | learned optimal advantage Ahat · 30 randomly generated gridworld MDPs; noiseless preferences | maximum action value distance from 0 | Helped |
| include transitions from absorbing states in preference segments (ablation) | greedy Ahat · 30 randomly generated gridworld MDPs; stochastic preferences | normalized mean return | Helped |
| include transitions from absorbing states in preference segments (ablation) | Ahat-as-reward · 30 randomly generated gridworld MDPs; stochastic preferences | normalized mean return | No clear effect |
| include transitions from absorbing states in preference segments (ablation) | learned optimal advantage Ahat · 30 randomly generated gridworld MDPs; stochastic preferences | maximum action value distance from 0 | Helped |
| use Ahat as the reward instead of the ground-truth reward | Q-learning · gridworld MDPs | area above the learning curve (AAC) | Helped |
| use the exact optimal advantage A* as the reward instead of the ground-truth reward | Q-learning · gridworld MDPs | area above the learning curve (AAC) | Helped |
| use the approximate optimal advantage Ahat as the reward instead of the ground-truth reward | policy improvement with Q-learning · gridworld MDPs | sample efficiency | Helped |
| use greedy action selection from Ahat instead of treating Ahat as a reward and applying policy improvement | Ahat-based policy learning · gridworld MDPs | performance | Mixed |
Surprising findings
- Learning the optimal advantage under the partial return assumption and using it as reward performs well, because it is highly shaped.
- Adding a constant to the learned advantage changes nothing in the learned likelihood, making the maximum value underspecified and able to change optimal policies in variable-horizon tasks.
Limitations
- Experiments are limited to small tabular gridworld MDPs with synthetic regret-based preferences.
- Learned advantage may have nonzero max per state, leading to termination/non-termination bias that is not fully characterized.
- Regret model requires a differentiable approximation of the optimal advantage, which is challenging for complex LLM tasks.
Experimental setup
Datasets: synthetic gridworld MDPs with regret-generated preferences
Benchmarks: randomly generated gridworld MDPs
Metrics: normalized mean return area above curve (AAC) max advantage per state
Baselines: partial return preference model greedy A*_r Q-learning with ground-truth reward r
Builds on: regret-based preference model (Knox et al.) potential-based reward shaping (Ng et al.) Christiano et al. preference learning Q-learning
Closest papers by meaning
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.91)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.90)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.90)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.89)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.