Research map / Reinforcement learning / RL from feedback
GUIDE: Real-Time Human-Shaped Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GUIDE, a real-time human-guided RL framework with continuous hover-based human feedback converted to dense rewards plus a learned feedback simulator for continued training, evaluated with a 50-subject human study on CREW tasks.
Problem
Real-time decision-making with sparse rewards and visual observations is hard; existing human-guided RL uses discrete, time-stepped feedback, relies on few subjects, and offers no way to keep improving once human feedback stops.
Approach
Continuous-valued human feedback given at every step is grounded into per-step dense rewards added to sparse environment rewards, and a neural feedback simulator trained concurrently on (state, action, feedback) pairs replaces the human for automated continued training.
Key results
- On Find Treasure and Hide-and-Seek, GUIDE-DDPG reached up to 30% higher success rate than its RL counterpart and about 50% higher than c-Deep TAMER with the same human guidance time.
- With only 10 minutes of human feedback, the approach achieves up to 30% increase in success rate over the RL baseline (abstract).
- Subjects with higher cognitive test scores produced better-performing agents (reported Pearson coefficient 7.522 with p=0.001, as stated in the paper).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GUIDE-DDPG instead of DDPG | DDPG · Find Treasure | success rate: up to 30 percent higher | Helped |
| Use GUIDE-DDPG instead of DDPG | DDPG · 1v1 Hide-and-Seek | success rate: up to 30 percent higher | Helped |
| Use GUIDE instead of c-Deep TAMER | c-Deep TAMER · Find Treasure and 1v1 Hide-and-Seek | success rate: 50 percent higher | Helped |
| Use GUIDE instead of c-Deep TAMER | c-Deep TAMER · Find Treasure and 1v1 Hide-and-Seek | success rate: up to 40 points | Helped |
| Use GUIDE instead of DDPG | DDPG · Bowling | score | No clear effect |
| Use GUIDE instead of DDPG | DDPG · Find Treasure exploration | visible area ratio | Mixed |
Surprising findings
- Average performance of all 50 untrained human trainers still surpassed the RL baseline by a large margin and c-Deep TAMER on the harder tasks.
- Reflex time was positively related to 1v1 Hide-and-Seek score, attributed to fast-paced dynamics needing quick feedback.
Experimental setup
Datasets: CREW environments (Bowling, Find Treasure, 1v1 Hide-and-Seek) human feedback collected from 50 subjects
Benchmarks: CREW Bowling CREW Find Treasure CREW 1v1 Hide-and-Seek
Metrics: bowling score success rate percentage of trainers reaching score milestones exploration visible-area ratio Pearson correlation with cognitive test scores
Baselines: DDPG SAC c-Deep TAMER heuristic dense reward
Builds on: Deep TAMER DDPG COACH interactive reward shaping
Closest papers by meaning
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.89)
- M³HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality (ICML 2025 · similarity 0.88)
- Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning (ICLR 2025 · similarity 0.88)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.88)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.88)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.