Research map / Reinforcement learning / Visual and representation RL
Reinforcement Learning from Imperfect Corrective Actions and Proxy Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICoPro, a value-based deep RL method that iteratively combines sparse human corrective actions (via margin loss) with imperfect proxy rewards (via RL losses plus margin loss on pseudo-labels from a target Q-network).
Problem
RL agents trained with imperfect proxy rewards suffer human-agent misalignment; the goal is to learn policies better aligned with human preferences using both proxy rewards and sparse, possibly imperfect, human corrective actions.
Approach
Iterate three phases (data collection with sparse corrective actions, a margin-loss finetune phase on labels, and a propagation phase combining TD/n-step RL losses with margin losses on both human labels and target-network pseudo-labels), so proxy rewards and corrective actions complement each other.
Key results
- On Atari with large label budget, ICoPro achieves the best mean scores in most games (e.g., Boxing 61.52 vs labeler 66.05, Battlezone 20030 vs Rainbow 17058), outperforming BC-L and HGDAgger-L baselines.
- ICoPro reaches the labeler's Pong score with 375 feedback actions and 375K environment steps, versus 10K labels and 3M steps for ActionImitation.
- In a real-human user study on highway with 500 labels, ICoPro-Human reached Speed-Avg 29.02 and %Crash 0.22 with PR4, a behavior Rainbow could not obtain with PR4 alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine corrective-action margin losses with proxy-reward RL losses | ICoPro · Atari games, large and small label budgets | raw episodic return | Helped |
| remove pseudo-labels (ablation) | ICoPro · Atari, limited feedback and challenging tasks | raw episodic return | Hurt |
| remove the Finetune phase (ablation) | ICoPro · Atari, limited feedback and challenging tasks | raw episodic return | Hurt |
| remove the Finetune phase (ablation) | ICoPro · Battlezone, small label budget | raw episodic return: 19969.09 -> 21253.33 | Helped |
| remove pseudo-labels (ablation) | ICoPro · Seaquest, large label budget | raw episodic return: 1274.76 -> 1213.98 | Hurt |
| use PVP loss instead of margin loss | ICoPro comparator PVP-I · Atari games | raw episodic return | Hurt |
| add proxy rewards to PVP-I | PVP-I · Atari games | raw episodic return | Mixed |
| add RL propagation losses using proxy rewards to corrective-action-only learning | ICoPro versus BC-L and HGDAgger-L · Atari games | raw episodic return | Helped |
| train with real corrective actions and proxy rewards rather than behavior cloning on those labels | ICoPro-Human versus BC-L · Highway | Speed-Avg: 27.90 -> 29.02 | Helped |
| train with real corrective actions and proxy rewards rather than behavior cloning on those labels | ICoPro-Human versus BC-L · Highway | %Crash: 0.41 -> 0.22 | Helped |
Surprising findings
- Pure proxy-reward Rainbow often converges to non-optimal performance, but combining it with sparse corrective actions recovers labeler-level performance.
- PVP-style fixed-bound losses performed worse when proxy rewards were added (PVP-IR worse than PVP-I in some games).
Limitations
- Uniform query selection; more advanced sampling could reduce feedback needs.
- No theoretical analysis of when proxy rewards and corrective actions synergize.
- Main experiments use simulated scripted labelers; real-human evaluation is limited to a small user study.
Experimental setup
Datasets: Atari (ALE) games highway-env driving environment
Benchmarks: Atari (Seaquest, Boxing, Battlezone, Frostbite, Alien, Hero, MsPacman, Pong, Freeway, Enduro) highway driving
Metrics: episodic raw reward %Crash Distance-Avg Speed-Avg %LaneChange-Avg
Baselines: Rainbow BC-L HGDAgger-L DQfD-I PVP-I PVP-IR
Builds on: Rainbow DQfD PVP HG-DAgger DERainbow
Closest papers by meaning
- Strategyproof Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.88)
- Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.88)
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization (NeurIPS 2025 · similarity 0.88)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.