Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIM, a robot-gated interactive imitation learning method that trains a proxy Q-function to learn when to request expert takeover, adaptively reducing interventions as the agent improves.
Problem
Robot-gated interactive imitation learning relies on fixed uncertainty thresholds that misalign with human intervention decisions and keep querying the expert even after the agent is proficient, imposing high expert monitoring cost.
Approach
Learn a proxy Q-function that is trained to be high on agent actions deviating from expert actions and low on expert actions, and use its quantile-based threshold as the switch-to-human criterion; the intervention rate shrinks automatically as the policy aligns with the expert, with no variance ensemble needed.
Key results
- In MetaDrive with 2K expert-involved steps, AIM achieves 0.82 success rate and 328.4 episodic return versus 0.58 for Thrifty-DAgger and 0.60 for Ensemble-DAgger.
- In MiniGrid Four Room, AIM reaches 0.63 success rate versus 0.42 for Thrifty-DAgger using 0.4K expert data versus 2K.
- AIM uses 1.9K expert data (0.24 intervention rate) in MetaDrive, comparable to PVP's 0.4K (0.19) expert data while not requiring continuous monitoring.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the adaptive proxy-Q intervention mechanism | robot-gated IIL · MetaDrive | success rate: 0.62 { $\pm$ 0.06} -> 0.82 { $\pm$ 0.06} | Helped |
| use the adaptive proxy-Q intervention mechanism | robot-gated IIL · MetaDrive | episodic return: 310.8 { $\pm$ 16.7} -> 328.4 { $\pm$ 20.4} | Helped |
| use the adaptive proxy-Q intervention mechanism | robot-gated IIL · MetaDrive | route completion: 0.78 { $\pm$ 0.07} -> 0.91 { $\pm$ 0.03} | Helped |
| use the adaptive proxy-Q intervention mechanism | robot-gated IIL · MiniGrid Four Room | success rate: 0.42 { $\pm$ 0.12} -> 0.63 { $\pm$ 0.05} | Helped |
| add Q-function labeling instead of reward labeling (ablation) | AIM · MetaDrive | success rate: 0.61 -> 0.82 | Helped |
| add Q-function labeling instead of reward labeling (ablation) | AIM · MetaDrive | episodic return: 275.1 -> 328.4 | Helped |
| add Q-function labeling instead of reward labeling (ablation) | AIM · MetaDrive | route completion: 0.82 -> 0.91 | Helped |
| add TD loss (ablation) | AIM · MetaDrive | success rate: 0.57 -> 0.82 | Helped |
| add TD loss (ablation) | AIM · MetaDrive | episodic return: 246.7 -> 328.4 | Helped |
| add TD loss (ablation) | AIM · MetaDrive | route completion: 0.75 -> 0.91 | Helped |
Surprising findings
- AIM attains higher alignment with the expert while requesting fewer interventions than both Ensemble-DAgger and Thrifty-DAgger.
- Uncertainty-based Thrifty-DAgger keeps querying the expert at the same locations even after the agent has mastered them.
Limitations
- Assumes the expert knows the optimal control strategy and behaves correctly.
- No real-human experiments or user studies; simulated neural experts stand in for humans.
- Extension to multi-agent human interaction is unexplored.
Experimental setup
Datasets: MetaDrive safety benchmark (50 training maps, 50 test maps) MiniGrid Four Room / multi-room
Benchmarks: MetaDrive safety benchmark MiniGrid Four Room
Metrics: success rate episodic return route completion rate expert data usage overall intervention rate
Baselines: Behavioral Cloning HG-DAgger PVP Ensemble-DAgger Thrifty-DAgger
Builds on: DAgger Thrifty-DAgger Ensemble-DAgger Proxy Value Propagation PPO-Lagrangian
Closest papers by meaning
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.89)
- When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning (NeurIPS 2026 · similarity 0.87)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.86)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.86)
- Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model (ICLR 2025 · similarity 0.85)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.