Research map / Reinforcement learning / RL from feedback
Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ada-Pref and Ada-DPO, a DRO-derived adaptive preference loss that learns a per-pair scaling factor tau controlling the logit-to-reward-difference mapping in reward learning and DPO, evaluated on PyBullet robot control and Llama-2 7B summarization/dialogue.
Problem
Standard Bradley-Terry cross-entropy reward learning assumes a single linear scaling between preference logits and reward differences for all pairs, which cannot capture varying preference strength and causes misalignment between reward model selection and downstream policy performance.
Approach
Derive a per-instance adaptive preference loss from KL-constrained DRO over binary preference distributions, yielding a learned scaling tau_i per pair (small for ambiguous, large for clear preferences), a strictly convex univariate problem solved by projected Newton, and extend it to DPO (Ada-DPO).
Key results
- On PyBullet with policy-based selection, Ada-Pref improves return over Pref on all three tasks (e.g., Hopper 1692.1 vs 1324.91; Ant 3177.11 vs 2917.81).
- When reward is selected by preference accuracy, Ada-Pref retains higher return than Pref (e.g., Hopper 1134.73 vs 744.66), indicating better reward-policy alignment.
- Ada-DPO achieves the highest win rate over DPO, IPO and SLiC-HF in summarization and dialogue when selected by win rate, and gains significantly over DPO when selected by accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard preference loss with adaptive preference scaling | Ada-Pref vs Pref · HalfCheetah, best-policy selection | return: 2724.42 -> 2875.45 | Helped |
| replace standard preference loss with adaptive preference scaling | Ada-Pref vs Pref · Ant, best-policy selection | return: 2917.81 -> 3177.11 | Helped |
| replace standard preference loss with adaptive preference scaling | Ada-Pref vs Pref · Hopper, best-policy selection | return: 1324.91 -> 1692.1 | Helped |
| replace standard preference loss with adaptive preference scaling | Ada-Pref vs Pref · HalfCheetah, best-reward-model selection | return: 2620.83 -> 2865.07 | Helped |
| replace standard preference loss with adaptive preference scaling | Ada-Pref vs Pref · Ant, best-reward-model selection | return: 2750.99 -> 3008.69 | Helped |
| replace standard preference loss with adaptive preference scaling | Ada-Pref vs Pref · Hopper, best-reward-model selection | return: 744.66 -> 1134.73 | Helped |
| replace DPO loss with adaptive preference scaling and quadratic regularization (ablation) | Ada-DPO-Quad vs DPO · Dialogue | win rate: 53.38 -> 56.00 | Helped |
| replace DPO loss with adaptive preference scaling and quadratic regularization (ablation) | Ada-DPO-Quad vs DPO · Dialogue | preference accuracy: 54.39 -> 53.56 | Hurt |
| use quadratic regularization for scaling factors (ablation) | Ada-Pref-Quad vs Pref · Ant | return: 2917.81 -> 3116.57 | Helped |
| use quadratic regularization for scaling factors (ablation) | Ada-Pref-Quad vs Pref · Ant | preference accuracy: 90.08 -> 90.66 | Helped |
| use adaptive preference scaling rather than the standard loss | Ada-Pref vs Pref · HalfCheetah, best-policy selection | preference accuracy: 89.09 -> 89.46 | No clear effect |
| use adaptive preference scaling rather than the standard loss | Ada-Pref vs Pref · Ant, best-policy selection | preference accuracy: 85.57 -> 85.48 | No clear effect |
Surprising findings
- Around 60% of learned scaling factors on the Ant task hit the upper bound, suggesting the upper bound strongly shapes the solution.
- Smaller values of rho generally performed better, and the regularization parameter rho was more important than the other hyperparameters.
Limitations
- Three extra hyperparameters (tau_0, tau_max, rho) require tuning, with rho most sensitive.
- Robotic experiments use synthetic preferences from ground-truth rewards rather than human labels.
- NLG evaluation relies on Claude 2 as judge; human evaluation not performed.
Experimental setup
Datasets: PyBullet HalfCheetah/Ant/Hopper synthetic preferences TL;DR summarization dataset TL;DR human preference dataset Anthropic Helpful and Harmless dialogue preferences
Benchmarks: PyBullet HalfCheetah PyBullet Ant PyBullet Hopper TL;DR summarization win rate Anthropic HH dialogue win rate
Metrics: return preference prediction accuracy win rate
Baselines: Pref (cross-entropy reward learning) DPO IPO SLiC-HF
Builds on: Bradley-Terry model direct preference optimization proximal policy optimization PEBBLE SLiC-HF
Closest papers by meaning
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.92)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.92)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.92)
- Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.91)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.