Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Efficient Active Imitation Learning with Random Network Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RND-DAgger, an active imitation learning method that uses a Random Network Distillation out-of-distribution measure on states (not expert actions) to trigger expert takeovers, reducing expert queries in racing, maze navigation, and HalfCheetah.
Problem
Active imitation learning for underspecified tasks such as realistic video game bots, where expert interventions are costly and existing DAgger variants require the expert to continuously monitor or provide actions at every step.
Approach
Use the prediction error between a fixed random target network and a trained predictor network as a state-based OOD measure to decide when the expert takes control, avoiding per-step expert action comparisons; add a minimal expert time (W consecutive in-distribution frames) before handing control back.
Key results
- On RaceCar, RND-DAgger uses 368 context switches vs 2785 for Ensemble-DAgger while reaching 0.944 success rate (Ensemble-DAgger 0.952).
- On 3D Maze, RND-DAgger achieves the highest success rate of 0.717 vs 0.626 for Ensemble-DAgger and 0.575 for Lazy-DAgger.
- With a human expert on RaceCar, RND-DAgger performs comparably to HG-DAgger while significantly lowering total expert time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use state-based RND novelty to trigger expert interventions instead of action discrepancy | active imitation learning · Maze | success rate: 0.626 ± 0.045 -> 0.717 ± 0.018 | Helped |
| use state-based RND novelty to trigger expert interventions instead of action discrepancy | active imitation learning · RaceCar | context switches: 2785 ± 41 -> 368 ± 11 | Helped |
| use state-based RND novelty to trigger expert interventions instead of action discrepancy | active imitation learning · HalfCheetah | cumulative reward: 2314 ± 278 -> 2490 ± 160 | Helped |
| add Minimal Expert Time (ablation) | RND-DAgger · HalfCheetah | context switches: 1985 ± 215 -> 708 ± 130 | Helped |
| add Minimal Expert Time (ablation) | RND-DAgger · HalfCheetah | cumulative reward: 2432 ± 175 -> 2490 ± 160 | Helped |
| use a smaller random-network architecture (ablation) | RND-DAgger · HalfCheetah | cumulative reward: 2490 ± 160 -> 2167 ± 140 | Hurt |
| remove historical context (ablation) | RND-DAgger · HalfCheetah | cumulative reward: 2490 ± 160 -> 2172 ± 290 | Hurt |
| add Minimal Expert Time (ablation) | Ensemble-DAgger · RaceCar | context switches: 2785 ± 41 -> 77 ± 4 | Helped |
| add Minimal Expert Time (ablation) | Ensemble-DAgger · RaceCar | success rate: 0.952 ± 0.018 -> 0.935 ± 0.019 | Hurt |
| add Minimal Expert Time (ablation) | Ensemble-DAgger · Maze | success rate: 0.626 ± 0.045 -> 0.707 ± 0.023 | Helped |
| add Minimal Expert Time (ablation) | Lazy-DAgger · HalfCheetah | context switches: 312 ± 45 -> 78 ± 0.0 | Helped |
| add Minimal Expert Time (ablation) | Lazy-DAgger · RaceCar | success rate: 0.939 ± 0.025 -> 0.943 ± 0.011 | No clear effect |
Surprising findings
- Lazy-DAgger achieves fewer context switches than RND-DAgger on HalfCheetah (312 vs 708), attributed to the oracle expert's consistent actions.
- Adding minimal expert time to Lazy-DAgger on HalfCheetah forces near-constant expert control, yielding poor performance.
Limitations
- Requires the expert to take control immediately, which may be impossible or dangerous in real-time environments.
- Not yet evaluated on partially observable or pixel-based environments; noisy-TV issues are a concern.
- Results with oracle experts in most experiments; only one human-expert study on RaceCar.
Experimental setup
Datasets: RaceCar expert traces 3D Maze navigation trajectories HalfCheetah trajectories from oracle agent
Benchmarks: RaceCar HalfCheetah 3D Maze
Metrics: success rate cumulative episode reward dataset size context switches total expert time
Baselines: BC DAgger Ensemble-DAgger Lazy-DAgger HG-DAgger
Builds on: DAgger Ensemble-DAgger Lazy-DAgger Random Network Distillation Goal-Conditioned Behavioral Cloning
Closest papers by meaning
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Mimicking Better by Matching the Approximate Action Distribution (ICML 2024 · similarity 0.87)
- Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration (ICLR 2025 · similarity 0.87)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.87)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.87)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.