Research map / Reinforcement learning / RL from feedback
Rating-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes rating-based RL (RbRL), which learns reward functions from absolute multi-class human ratings of individual trajectory segments via a new multi-class cross-entropy loss, and compares it to PbRL in synthetic and human experiments.
Problem
Reward learning from human feedback in RL relies on pairwise preferences, which give only one bit per comparison, lack absolute quality information, and struggle with similar or incomparable trajectories.
Approach
Replace pairwise preference queries with absolute multi-class ratings of single trajectory segments; model class probabilities from normalized predicted returns with Gaussian-like class-boundary functions and fit boundaries so class proportions match the human-labeled data, optimized with a multi-class cross-entropy loss.
Key results
- On synthetic Walker, RbRL with n=5,6 outperforms PbRL, and performance increases with more rating classes; on Quadruped, RbRL with n=2,3 outperforms PbRL but more classes degrade performance.
- In the human study with 20 participants, participants gave about 14.03 ratings per minute versus 8.7 preferences per minute, so preferences took about 62% more time.
- In the non-expert user study, RbRL performs similarly to or better than PbRL on Cheetah and Swimmer, and questionnaires indicate users find RbRL less demanding and discouraging than PbRL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RbRL with 5 or 6 rating classes instead of PbRL | RbRL · Walker synthetic experiment | mean reward | Helped |
| Use RbRL with 2 or 3 rating classes instead of PbRL | RbRL · Quadruped synthetic experiment | mean reward | Helped |
| Increase the number of rating classes beyond 3 in RbRL (ablation) | RbRL · Quadruped synthetic experiment | mean reward | Hurt |
| Use 3–7 rating classes rather than 2 or 8 (ablation) | RbRL · Cheetah MuJoCo expert experiment | performance | Helped |
| Use RbRL instead of PbRL | Human user study · Cheetah and Swimmer, non-expert participants | environment reward | Mixed |
| Use RbRL instead of PbRL | Human user study · Cheetah and Swimmer, expert trials | environment reward | Helped |
| Use RbRL instead of PbRL | Human user study · Cheetah and Swimmer, top 3 non-expert participants | mean reward | Helped |
| Use RbRL instead of PbRL | Human user study · Hopper backflip task | learned behavior quality | Helped |
| Request individual ratings rather than pairwise preferences | Human feedback collection · Cheetah, Swimmer, and Hopper user study | queries per minute: 14.03 ratings per minute; 8.7 preferences per minute | Helped |
| Request pairwise preferences rather than individual ratings | Human feedback collection · Human user study | time per query: 62% more time | Hurt |
| Use RbRL instead of PbRL | Human user study participants · Post-experiment questionnaire | perceived difficulty | Helped |
Surprising findings
- For Quadruped, increasing the number of rating classes beyond 3 decreased RbRL performance, which the authors attribute possibly to the boundary selection used for synthetic feedback.
- Participants preferred RbRL and PbRL equally when asked directly for a preference, and one participant preferred PbRL because it was more challenging.
Limitations
- Class imbalance in human ratings can negatively affect reward learning.
- RbRL learns a less refined reward function because it does not distinguish samples within the same rating class.
- Human ratings may be inconsistent over time due to attention and fatigue; the paper does not address this.
Experimental setup
Datasets: Walker (MuJoCo) Quadruped Cheetah (MuJoCo) Swimmer (MuJoCo) Hopper (MuJoCo)
Benchmarks: Walker Quadruped Cheetah Swimmer Hopper
Metrics: mean ground-truth reward learned backflip behavior (qualitative video) questionnaire responses number of queries per 30 minutes
Baselines: PbRL (preference-based RL) PEBBLE SURF
Builds on: preference-based reinforcement learning PEBBLE SURF TAMER
Closest papers by meaning
- Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models (ICML 2025 · similarity 0.87)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.85)
- GUIDE: Real-Time Human-Shaped Agents (NeurIPS 2024 · similarity 0.84)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.84)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.84)
- Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.