Research map / Reinforcement learning / Imitation learning
OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OLLIE, an offline-to-online imitation learning method that learns a near-expert offline policy plus a discriminator aligned with it, so GAIL finetuning starts without unlearning pretrained knowledge.
Problem
Offline-to-online imitation learning: naively pretraining a policy offline and finetuning with GAIL degrades performance because the initial GAIL discriminator is random and misaligned with the pretrained policy, causing unlearning of pretraining knowledge.
Approach
Derives a convex-concave stochastic saddle point objective from a reverse-KL-regularized IL surrogate that is optimized fully offline; the optimal dual variable y* yields a policy via weighted behavior cloning and, by stitching the offline discriminator d* with y*, a discriminator aligned with that policy at no extra cost.
Key results
- OLLIE outperforms baselines in 20/24 offline IL settings with varying imperfect data quality, often by 2-4x with fewer expert trajectories.
- During online finetuning OLLIE improves offline performance by about 2x within fewer than 10 environment episodes, avoiding the unlearning seen with naive GAIL finetuning.
- On vision-based can and square tasks where GAIL from scratch fails, OLLIE still performs well.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OLLIE instead of the strongest reported baseline | offline IL · Ant (random) | normalized score: 39.3±9.0 -> 57.1±7.0 | Helped |
| Use OLLIE instead of the strongest reported baseline | offline IL · HalfCheetah (med.-rep.) | normalized score: 87.5±3.0 -> 44.8±4.1 | Hurt |
| Use OLLIE instead of the strongest reported baseline | offline IL · HalfCheetah (medium) | normalized score: 83.6±3.2 -> 62.3±4.2 | Hurt |
| Use OLLIE instead of the strongest reported baseline | offline IL · Walker2d (medium) | normalized score: 92.9±3.7 -> 111.7±0.9 | Helped |
| Use OLLIE instead of the strongest reported baseline | offline IL · Hammer (human) | normalized score: 16.7±8.9 -> 46.1±6.5 | Helped |
| Use OLLIE instead of the strongest reported baseline | offline IL · Pen (human) | normalized score: 49.5±2.9 -> 67.4±5.6 | Helped |
| Use OLLIE instead of the strongest reported baseline | offline IL · Door (human) | normalized score: 24.8±3.6 -> 28.9±2.9 | Helped |
| Use OLLIE instead of the strongest reported baseline | offline IL · Relocate (clone) | normalized score: 10.2±4.4 -> 40.8±6.9 | Helped |
| Initialize GAIL's discriminator aligned with the pretrained policy rather than randomly | GAIL finetuning · offline-to-online IL | performance | Helped |
| Pretrain with OLLIE and finetune online with GAIL | OLLIE · online finetuning | normalized performance: 2x; fewer than 10 environmental episodes | Helped |
| Pretrain with OLLIE rather than training GAIL from scratch | vision-based can and square · online finetuning | performance | Helped |
Surprising findings
- Pretrained policies from several offline IL methods (DWBC, MLIRL, ISWBC) degrade at the start of GAIL finetuning, so offline pretraining can hurt online IL without discriminator alignment.
Limitations
- Finetuning relies on GAIL, which is on-policy and adversarial, so it may be sample-inefficient and unstable.
- Method is not directly compatible with off-policy or non-adversarial IL finetuning in the current form.
Experimental setup
Datasets: D4RL AntMaze D4RL MuJoCo D4RL Adroit D4RL FrankaKitchen robomimic vision-based MuJoCo
Benchmarks: D4RL AntMaze D4RL MuJoCo D4RL Adroit D4RL FrankaKitchen robomimic vision-based MuJoCo
Metrics: normalized score demonstration efficiency convergence speed
Baselines: BC NBCU CSIL DWBC MLIRL ISWBC GAIL OptiDICE CQL
Builds on: GAIL DemoDICE ISWBC MARWIL SAC
Closest papers by meaning
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.90)
- SEABO: A Simple Search-Based Method for Offline Imitation Learning (ICLR 2024 · similarity 0.90)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.90)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.90)
- Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model (ICLR 2025 · similarity 0.90)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.