Research map / Reinforcement learning / RL from feedback
Reward Learning through Ranking Mean Squared Error
Chaitanya Kharyal, Calarina Muslimani, Matthew E. Taylor
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the RbRL cross-entropy loss with the ranking MSE (rMSE) loss (ablation) | SAC with learned reward · offline feedback: Reacher, Inverted Double Pendulum, Half Cheetah | SAC return / learning speed: faster learning or higher final return in all domains (p < 0.03) | Improved |
| use R4 instead of RbRL and preference methods (PEBBLE, SURF, QPA) in online feedback | SAC on DMC · six DMC environments | SAC return: faster learning in 3 of 6 environments, higher final return in 4 of 6 (p < 0.0125) | Improved |
| use R4 instead of RbRL on real human ratings | R4 reward learning · Reacher (human ratings) | final return: -44.62 -> -27.65 | Improved |
| use R4 instead of RbRL on real human ratings | R4 reward learning · Hopper (human ratings) | final return: 723.87 -> 1410.10 | Improved |
| train R4 on human ratings instead of simulated ratings | R4 · Reacher | SAC return: small gap | Hurt |
| remove dynamic feedback schedule and stratified sampling from R4 (ablation) | R4 online · three online DMC domains | SAC return | No clear effect |
| give baselines the dynamic feedback schedule (ablation) | RbRL, PEBBLE, SURF, QPA · online DMC | SAC return | Hurt |
| inject label noise into ratings (reassign eta% of trajectories to true bin +/-1) (ablation) | R4 vs RbRL · Inverted Double Pendulum (offline) | SAC return: R4 at 80% noise comparable to RbRL at 10% noise | Mixed |
| vary the number of rating bins (ablation) | R4 vs RbRL · Reacher | SAC return | No clear effect |
| vary soft-rank regularization strength (ablation) | R4 · Inverted Double Pendulum | final return: 83% of runs with strengths 0.065 to 1 succeed | No clear effect |
Closest papers by meaning
- Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models (ICML 2025 · similarity 0.90)
- Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.89)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.89)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
