Research map / Reinforcement learning / Imitation learning
Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an oracle-efficient, agnostic interactive imitation learning algorithm (MFTPL-P) with regret guarantees for general discrete policy classes and a practical bootstrap-ensemble variant (BD) that outperforms DAgger and BC on MuJoCo tasks.
Problem
Interactive imitation learning in the agnostic setting (expert not contained in the policy class) requires efficient algorithms with finite-sample guarantees for general policy classes, while prior provably efficient methods need strong assumptions such as small separator sets or convex losses.
Approach
Reduce interactive IL to no-regret online learning and approximate an in-expectation Follow-the-Perturbed-Leader using an ensemble of oracle calls trained on the aggregated expert data plus Poisson-sized perturbation samples drawn from a covering distribution; a practical variant replaces perturbations with bootstrap resampling of the dataset.
Key results
- MFTPL-P (with fixed-size perturbations) and BD with E=25 outperform DAgger and BC in continuous control under both realizable and non-realizable expert settings, with the gap most evident in non-realizable settings.
- Theorem gives regret O~(sqrt(N)(A(lnB)^2/(sigma K^2))^{1/4} + ...), implying an expert-demonstration sample complexity of O~(mu^2 H^2 sqrt(A) lnB / (eps^2 sqrt(sigma))) versus H^4/eps^2 for BC.
- BD with E=5 consistently improves over E=1 and approaches E=25 performance; BD with E=25 runs about 5x longer than BD with E=5, DAgger and BC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sample-based perturbations with an appropriate perturbation size (ablation) | MPX · Ant and Hopper | average reward | Helped |
| Use perturbation states collected by DAgger instead of uniform state-space samples (ablation) | MPX · Ant and Hopper | average reward | Helped |
| Increase bootstrap ensemble size from E=1 to E=5 (ablation) | Bagging · Ant, Half-Cheetah, Hopper, and Walker2D | average reward | Helped |
| Increase bootstrap ensemble size from E=5 to E=25 (ablation) | Bagging · Continuous control tasks | average reward | Helped |
| Use bootstrap ensemble size E=1 instead of DAgger (ablation) | Bagging · Continuous control tasks | average reward | Hurt |
| Increase bootstrap ensemble size from E=5 to E=25 (ablation) | Bagging · Realizable Hopper | average reward | No clear effect |
| Train the final policy with naive supervised learning rather than bagging (ablation) | Bagging data-collection method · Ant and Hopper | average reward | No clear effect |
| Use an ensemble for data collection rather than DAgger data collection (ablation) | Bagging · Ant and Hopper | average reward | Helped |
| Use the MPF perturbation method rather than DAgger or BC | MPF · Four continuous control tasks | average reward | Helped |
| Use bootstrap ensemble training rather than DAgger | Bagging · Four continuous control tasks | average reward | Helped |
| Use bootstrap ensemble size E=5 rather than E=25 (ablation) | Bagging · Realizable settings | running time | Helped |
Surprising findings
- Using ensembles for data collection, not the final bagging, drives most of the gain of BD over DAgger; naive supervised learning on BD-collected data matches BD performance.
- A constructed example shows that any no-regret policy sequence in the reduction framework can converge to a globally suboptimal policy with linear cumulative cost gap.
Limitations
- Guarantees are runtime-dependent: the policy class bias term min over the mixed policy class is not controlled in general.
- Requires sample access to a covering distribution d0 for the main algorithm, and the experiments' d0 (DAgger states) gives it an unfair advantage; the practical variant lacks finite-sample guarantees.
- Experiments use only 4 MuJoCo tasks with 25-trajectory evaluation and fixed hyperparameters.
Experimental setup
Datasets: OpenAI Gym MuJoCo tasks (Ant, Half-Cheetah, Hopper, Walker2D) with expert rollouts from TRPO-pretrained MLP experts
Benchmarks: Ant Half-Cheetah Hopper Walker2D
Metrics: average episode reward vs number of expert annotations running time GPU memory
Baselines: DAgger Behavior Cloning
Builds on: DAgger Follow the perturbed leader smoothed online learning Ross and Bagnell reduction to online learning
Closest papers by meaning
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.91)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learning (NeurIPS 2025 · similarity 0.89)
- Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model (ICLR 2025 · similarity 0.89)
- Causal Imitation for Markov Decision Processes: a Partial Identification Approach (NeurIPS 2024 · similarity 0.89)
- Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.