Research map / Reinforcement learning / RL from feedback
Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ERL-VLM, which queries Gemini-1.5-Pro for absolute Likert ratings of trajectory segments and trains rating-based RL reward models with stratified sampling, weighted loss, and MAE loss to learn rewards from image observations and task language.
Problem
Learning reward functions from VLM feedback is sample-inefficient and ambiguous when VLMs compare pairs of trajectories, and existing rating-based RL is unstable under class imbalance and noisy VLM labels.
Approach
Query a VLM for absolute ratings (Likert scale) of individual trajectory segments rather than pairwise preferences, and make rating-based reward learning robust via stratified sampling with class-weighted loss and an MAE objective instead of cross-entropy.
Key results
- ERL-VLM outperforms all baselines on 6 of 7 simulated tasks, and on ALFRED it significantly outperforms RL-VLM-F, which nearly fails to learn.
- On ALFRED PickupObject and PutObject tasks, ERL-VLM outperforms even the sparse environment reward.
- On real robot, ERL-VLM reaches 0.73 (Sweep Bowl), 0.60 (Drawer Open), 0.47 (Pickup Banana) success vs 0.57/0.37/0.30 for sparse rewards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ERL-VLM rating-based rewards instead of the compared VLM-based reward methods | RL agents on MetaWorld and ALFRED · MetaWorld and ALFRED tasks | success rate | Helped |
| use ERL-VLM rating-based rewards instead of RL-VLM-F preference-based rewards | RL agents · Drawer Open | success rate | No clear effect |
| use ERL-VLM rating-based rewards instead of sparse environment rewards | ALFRED agents · PickupObject and PutObject | success rate | Helped |
| use ERL-VLM reward learning instead of BC or Sparse Rewards | offline IQL on the real robot · Sweep Bowl | success rate | Helped |
| use ERL-VLM reward learning instead of BC or Sparse Rewards | offline IQL on the real robot · Drawer Open | success rate | Helped |
| use ERL-VLM reward learning instead of BC or Sparse Rewards | offline IQL on the real robot · Pickup Banana | success rate | Helped |
| add MAE loss instead of cross-entropy loss (ablation) | rating-based RL reward learning · MetaWorld tasks | success rate | Helped |
| add stratified sampling and weighted loss (ablation) | rating-based RL reward learning · Sweep Into and Drawer Open | success rate | Helped |
| replace MAE with cross-entropy using label smoothing (ablation) | ERL-VLM reward learning · Drawer Open | success rate | No clear effect |
| increase the number of rating classes to 4 (ablation) | ERL-VLM · Sweep Into and Drawer Open | success rate | Hurt |
| use 2 rating classes instead of 3 (ablation) | ERL-VLM · Sweep Into | success rate | Helped |
| use 3 rating classes instead of 2 (ablation) | ERL-VLM · Drawer Open | success rate | Helped |
Surprising findings
- Label smoothing, a common remedy for noisy preference labels, was ineffective for multi-class VLM ratings.
- Increasing the number of rating classes to 4 degraded performance, while 2 classes worked best for Sweep Into and 3 for Drawer Open.
Limitations
- Relies on closed-source Gemini-1.5-Pro and its biases and hallucinations for feedback quality.
- Only visual observations and descriptive actions are used; no additional sensing modalities.
- Rating quality depends heavily on the prompt's analysis stage; rating descriptions are hand-designed.
Experimental setup
Datasets: MetaWorld ALFRED real-robot Sawyer demonstrations
Benchmarks: MetaWorld (Sweep Into, Drawer Open, Soccer) ALFRED (20 tasks, 10 kitchen scenes) real Sawyer robot (Sweep Bowl, Drawer Open, Pickup Banana)
Metrics: task success rate
Baselines: CLIP Score RoboCLIP Score RL-VLM-F Environment Reward Vanilla RbRL BC Sparse Rewards
Builds on: rating-based RL (RbRL) RL-VLM-F PEBBLE SAC IQL
Closest papers by meaning
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.91)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.89)
- FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning (ICML 2024 · similarity 0.88)
- Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.