Research map / Reinforcement learning / Imitation learning
Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that with per-state annotation cost, interactive DAgger-style imitation (StateDagger) and a hybrid offline+interactive method (WarmStartDagger) provably beat behavior cloning, with MuJoCo experiments.
Problem
Prior sharp analyses show behavior cloning cannot be improved in general when annotation cost is counted per trajectory, leaving little room for interactive imitation learning; the question is whether interaction helps when annotation cost is counted per state and how offline and interactive data ca
Approach
Measure annotation cost per queried state rather than per trajectory and run a one-sample-per-round DAgger variant (StateDagger) using exponential-weights online learning over each-step mixtures; hybrid WarmStartDagger restricts the policy class to policies consistent with offline demonstrations and then runs the same interactive loop.
Key results
- StateDagger matches or exceeds behavior cloning with 50% fewer state-wise annotations on the MuJoCo tasks at C=1 and is more stable.
- In the constructed MDP, behavior cloning needs Omega(N_1) offline trajectories, StateDagger needs Omega(H N_0) interactive annotations, while WarmStartDagger reaches expert performance with O(1) interactions and total cost about O(N_0 + C).
- WarmStartDagger with well-chosen offline data matches StateDagger in sample efficiency and outperforms both baselines on 4 MuJoCo tasks when C=2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use interactive state-wise annotations instead of offline Behavior Cloning | StateDagger vs. Behavior Cloning · MuJoCo continuous-control tasks | annotations at matched performance: 50% fewer annotations | Helped |
| Use interactive state-wise annotations instead of offline Behavior Cloning | StateDagger vs. Behavior Cloning · Walker2D | cost efficiency: C = 3 | Helped |
| Combine offline demonstrations with interactive state-wise annotations | WarmStartDagger vs. single-source methods · Ant | return: C = 1 | Helped |
| Combine offline demonstrations with interactive state-wise annotations | WarmStartDagger vs. baselines · MuJoCo continuous-control tasks | cost efficiency: C = 2 | Helped |
| Warm-start interactive learning with 800 offline state-action pairs | WarmStartDagger · Synthetic MDP | sample efficiency: 800 offline (state, expert action) pairs | Helped |
| Warm-start with 3200 offline state-action pairs rather than 200 (ablation) | WarmStartDagger · Synthetic MDP | return: 200; 3200 offline (state, expert action) pairs | Helped |
Surprising findings
- Hybrid WarmStartDagger is shown to strictly beat both pure offline and pure interactive methods in a constructed MDP, not just be a fallback.
Limitations
- Guarantees are for discrete actions with deterministic realizable experts, while MuJoCo experiments are continuous control, leaving a gap.
- Learned policies are bottlenecked by expert performance; the method does not aim to exceed the expert.
- Cost model assumes per-state queries; practical expert interventions over segments may be cheaper and are not modeled.
Experimental setup
Datasets: MuJoCo Hopper MuJoCo Ant MuJoCo HalfCheetah MuJoCo Walker2D
Benchmarks: MuJoCo Hopper MuJoCo Ant MuJoCo HalfCheetah MuJoCo Walker2D synthetic recoverable-state MDP
Metrics: expected return state-wise annotation efficiency state coverage
Baselines: Behavior Cloning StateDagger trajectory-wise DAgger
Builds on: DAgger Behavior Cloning AggreVate Hybrid reinforcement learning
Closest papers by meaning
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.92)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.91)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.90)
- OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning (ICML 2024 · similarity 0.90)
- A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.