Research map / Reinforcement learning / RL from feedback
Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDP, which pre-trains a reward model on reward-free random-policy transitions pseudo-labeled with the minimum reward, then runs standard scalar- or preference-based human-in-the-loop RL to improve feedback efficiency.
Problem
Human-in-the-loop RL methods that learn reward functions from human feedback require many human queries, limiting feedback efficiency.
Approach
Pseudo-label readily available reward-free sub-optimal transitions with the minimum environment reward, use them to pre-train the reward model via MSE regression and to seed the agent's replay buffer, so the reward model starts by assigning low reward to low-quality behavior before any human feedback.
Key results
- SDP significantly improved learning (final performance or AUC) in 23 of 36 preference-based experiments across four algorithms and nine environments, and never statistically hurt performance.
- SDP + R-PEBBLE significantly outperformed R-PEBBLE and Deep TAMER in scalar-feedback DMControl tasks and reached comparable final performance to SAC with as few as 60 feedback queries.
- In a 16-participant human study, SDP maintained significant gains over PEBBLE in AUC or final performance on Pendulum-swingup and Cartpole-swingup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SDP (sub-optimal data pre-training and agent update) to PEBBLE | PEBBLE with human preference feedback · Pendulum-swingup and Cartpole-swingup | AUC or final performance | Helped |
| Add SDP to preference-based RL algorithms | PEBBLE, RUNE, SURF, and MRN · Nine DMControl and Meta-World environments | final performance or AUC: 23 out of the 36 experiments | Helped |
| Add SDP to preference-based RL algorithms | PEBBLE, RUNE, SURF, and MRN · Nine DMControl and Meta-World environments | performance: remaining experiments | No clear effect |
| Add SDP to scalar feedback RL | R-PEBBLE · Walker-walk, Cheetah-run, and Quadruped-walk | final performance or AUC | Helped |
| Add SDP to scalar feedback RL | R-PEBBLE · Walker-walk, Cheetah-run, and Quadruped-walk | final performance: 60 feedback queries | No clear effect |
| Use related-task sub-optimal data instead of target-task sub-optimal data | SDP + PEBBLE · Walker-walk, Quadruped-run, and Door-open | performance | No clear effect |
| Use only reward-model pre-training, omitting the agent update phase (ablation) | SDP + R-PEBBLE · Walker-walk | performance | Hurt |
| Use only the agent update phase, omitting reward-model pre-training (ablation) | SDP + R-PEBBLE · Walker-walk | performance | Hurt |
| Increase feedback budget across 60, 100, and 200 queries (ablation) | SDP versus R-PEBBLE · Walker-walk | performance: 60, 100, 200 | Helped |
| Increase sub-optimal transitions from 5000 to 15000 or 50000 (ablation) | SDP + R-PEBBLE · Walker-walk | performance: 5000, 15000, 50000 | Helped |
| Change participant background, gender, age, or education group | SDP with human preference feedback · Pendulum-swingup and Cartpole-swingup | AUC or final performance | No clear effect |
Surprising findings
- Directly applying the zero-label offline RL sub-optimal data approach to human-in-the-loop RL was ineffective, per the authors.
- Fewer than 50% of surveyed preference learning studies tested algorithms with non-author human participants.
Limitations
- Relies on the assumption that random or low-quality trajectories have rewards close to the minimum, which may not hold in all tasks.
- Human study is small (16 participants) and limited to two DMControl tasks.
- Requires a known minimum environment reward for pseudo-labeling.
Experimental setup
Datasets: DMControl Suite (Walker-walk, Cheetah-run, Quadruped-walk, Cartpole-swingup, Pendulum-swingup) Meta-World (Hammer, Door-unlock, Door-lock, Drawer-open, Window-open) Random-policy sub-optimal transitions
Benchmarks: DMControl Suite Meta-World
Metrics: final performance (undiscounted return) area under the return curve (AUC) success rate
Baselines: PEBBLE RUNE SURF MRN SAC R-PEBBLE Deep TAMER
Builds on: PEBBLE SAC Deep TAMER SURF RUNE
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.90)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.90)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.90)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.89)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.