Research map / Reinforcement learning / RL from feedback
RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RIME, a PbRL method that combines a KL-divergence-based sample-selection denoising discriminator (with label flipping) and a reward-model warm start from intrinsic rewards to learn robustly from noisy preferences.
Problem
Preference-based RL degrades severely when human preference labels are noisy or corrupted, and existing PbRL methods focus on feedback efficiency at the cost of robustness.
Approach
Filters noisy preference labels online using a dynamic lower bound on the KL divergence between predicted preferences and annotated labels (with an uncertainty term for RL distribution shift), flips labels above a fixed upper bound, and warm-starts the reward model (and Q-network) by fitting intrinsic rewards during pre-training to avoid the pre-training-to-online transition gap.
Key results
- Across five error rates, RIME averages 837.79 on Walker, 415.52 on Quadruped and 85.70 on Button Press, outperforming PEBBLE (692.05, 208.66, 50.07) and other PbRL baselines.
- On Walker-walk with 30% noisy preferences, ablation shows full RIME reaches 741 vs 431 for PEBBLE; on Button Press with 30% noise it reaches 80.0 vs 22.0.
- With real non-expert human labels on Hopper (about 40% error), RIME significantly outperforms PEBBLE and performs consecutive backflips with 100 feedback.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RIME instead of PEBBLE | PEBBLE · Walker-walk, Cheetah-run, Quadruped-walk, Button Press, Sweep Into, and Hammer; five error rates | episode return or success rate: Walker: 692.05 ± 192.67 -> 837.79 ± 133.49; Cheetah: 604.77 ± 126.63 -> 602.18 ± 96.10; Quadruped: 208.66 ± 106.81 -> 41 | Helped |
| Add warm start to the reward model (ablation) | RIME · Walker-walk, ε=0.3 | episode return: 457 ± 190 -> 741 ± 139 | Helped |
| Add warm start to the reward model (ablation) | RIME · Button Press, ε=0.3 | success rate: 64.7 ± 26.5 -> 80.0 ± 27.7 | Helped |
| Add warm start to the reward model (ablation) | RIME · Button Press, ε=0.1 | success rate: 97.2 ± 4.6 -> 99.9 ± 0.3 | Hurt |
| Add lower-bound filtering τ_lower (ablation) | RIME · Walker-walk, ε=0.3 | episode return: 492 ± 188 -> 741 ± 139 | Helped |
| Add lower-bound filtering τ_lower (ablation) | RIME · Button Press, ε=0.3 | success rate: 35.1 ± 24.1 -> 80.0 ± 27.7 | Helped |
| Add upper-bound label flipping τ_upper (ablation) | RIME · Walker-walk, ε=0.3 | episode return: 693 ± 167 -> 741 ± 139 | Helped |
| Add upper-bound label flipping τ_upper (ablation) | RIME · Button Press, ε=0.3 | success rate: 51.4 ± 30.0 -> 80.0 ± 27.7 | Helped |
| Replace fixed τ_lower=0.5 with dynamic τ_lower (ablation) | RIME · Walker-walk, ε=0.3 | episode return: 98 ± 92 -> 741 ± 139 | Helped |
| Replace fixed τ_lower=0.5 with dynamic τ_lower (ablation) | RIME · Button-press, ε=0.3 | success rate: 36.8 ± 45.4 -> 80.0 ± 27.7 | Helped |
Surprising findings
- Label smoothing almost fails to handle corrupted labels in PbRL, while sample selection methods work better.
- RIME fails at error rates of 40% and 45% on Walker-walk and Cheetah-run even with ten times the minimum feedback.
Limitations
- Trade-off between sample efficiency and robustness: more feedback is needed as error rate increases.
- Several hyperparameters in the denoising discriminator require extra tuning.
- Simulated noise models (BPref) may not match real human preference noise; fails at error rates of 40% and above.
Experimental setup
Datasets: Meta-world Button Press Meta-world Sweep Into Meta-world Hammer DMControl Walker-walk DMControl Cheetah-run DMControl Quadruped-walk Hopper backflip (non-expert human labels)
Benchmarks: Meta-world Button Press Meta-world Sweep Into Meta-world Hammer DMControl Walker-walk DMControl Cheetah-run DMControl Quadruped-walk
Metrics: episode return success rate
Baselines: SAC (ground-truth reward) PEBBLE SURF RUNE MRN ADT MAE t-CE label smoothing
Builds on: PEBBLE SAC BPref mistake teacher adaptive denoising training
Closest papers by meaning
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.87)
- PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation (ICML 2024 · similarity 0.87)
- Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.87)
- Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Provable Offline Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.