Research map / Reinforcement learning / Imitation learning
RLIF: Interactive Imitation Learning as Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLIF, which runs an off-policy RL algorithm (RLPD) using human intervention events as negative rewards, so learning can exceed a suboptimal interventing expert under DAgger-like assumptions.
Problem
Interactive imitation learning (DAgger-style) assumes near-optimal expert interventions and cannot improve beyond the expert; the goal is to learn from interventions when the expert is suboptimal, without a task reward.
Approach
Treat the intervention event itself as a signal: label the transition leading to an intervention with reward -1 (0 otherwise) and run off-policy RL on all aggregated data, so the policy learns to avoid states that trigger interventions rather than copying the expert's possibly suboptimal actions.
Key results
- RLIF with value-based intervention averages 77.79 on adroit-pen versus 40.93 for DAgger and 54.00 for HG-DAgger across expert levels.
- On walker2d RLIF with value-based intervention averages 103.83 versus 36.75 for HG-DAgger and 30.27 for DAgger; for a 20% expert it reaches 102.85.
- On the real Franka robot RLIF reaches 100% success on peg insertion within six rounds (about 20 minutes) and 95% on cloth unfolding in seven rounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLIF with value-based interventions instead of the strongest reported DAgger-style baseline | RLPD-based RLIF · Adroit-Pen average | success rate: 54±5.81 -> 77.79±8.89 | Helped |
| Use RLIF with value-based interventions instead of the strongest reported DAgger-style baseline | RLPD-based RLIF · Walker2d average | normalized return: 67.36±4.06 -> 103.83±4.48 | Helped |
| Use RLIF with value-based interventions instead of the strongest reported DAgger-style baseline | RLPD-based RLIF · Hopper average | normalized return: 51.32±3.32 -> 108.53±0.47 | Helped |
| Use value-based rather than random interventions | RLIF · Adroit-Pen, Walker2d, and Hopper | performance: 77.79±8.89 vs 35.11±9.20; 103.83±4.48 vs 67.08±43.83; 108.53±0.47 vs 90.93±15.7 | Mixed |
| Use a 10% expert's rather than a 110% expert's Q reference (ablation) | RLIF · Walker2d, expert ~110% | normalized return: 108.99±5.28 -> 78.9±49.97 | Hurt |
| Use a 10% expert's rather than a 110% expert's Q reference (ablation) | RLIF · Walker2d, expert ~40% | normalized return: 99.66±5.9 -> 38.78±20.3 | Hurt |
| Use a 10% expert's rather than a 110% expert's Q reference (ablation) | RLIF · Walker2d, expert ~20% | normalized return: 102.85±2.26 -> 33.55±9.63 | Hurt |
| Use a 60% rather than a 110% expert's Q reference (ablation) | RLIF · Walker2d, expert ~110% | normalized return: 108.99±5.28 -> 85.91±19.31 | Hurt |
| Use a 60% rather than a 110% expert's Q reference (ablation) | RLIF · Walker2d, expert ~40% | normalized return: 99.66±5.9 -> 71.21±21.96 | Hurt |
| Use a 60% rather than a 110% expert's Q reference (ablation) | RLIF · Walker2d, expert ~20% | normalized return: 102.85±2.26 -> 66.8±8.22 | Hurt |
| Train from human intervention feedback on the real peg-insertion task | RLIF · Peg insertion, real robot | success rate: 100% | Helped |
| Train from human intervention feedback on the real cloth-unfolding task | RLIF · Cloth unfolding, real robot | success rate: 95% | Helped |
Surprising findings
- RLIF with random interventions performs poorly and is highly variable (e.g., 67.08 ± 43.83 on walker2d), indicating the intervention-timing model matters a lot.
Limitations
- Requires an RL algorithm that can train on mixed on-policy and suboptimal expert data, which is not trivially easy.
- Still requires online deployment with expert oversight, raising safety concerns.
- Theoretical guarantees rely on an assumed intervention model (beta > 0.5, value-based interventions) that may not match real humans.
Experimental setup
Datasets: D4RL pen-expert-v1 D4RL hopper-expert-v2 D4RL walker2d-expert-v2 manually collected suboptimal robot trajectories
Benchmarks: Adroit pen (sparse) Walker2d Hopper real-world Franka peg insertion real-world cloth unfolding
Metrics: normalized return success rate
Baselines: DAgger HG-DAgger BC IQL
Builds on: DAgger HG-DAgger RLPD soft actor-critic LCB-VI
Closest papers by meaning
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.90)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.90)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.90)
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (AAAI 2025 · similarity 0.90)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.