Research map / Reinforcement learning / Imitation learning
A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAILOR, a learning-to-search imitation method that learns a world model, reward model and critic from demonstrations plus base-policy rollouts, and plans residual corrections via MPPI at test time, with expert-iteration distillation into the diffusion base policy.
Problem
Behavioral cloning policies fail after leaving the support of expert demonstrations and cannot recover; the goal is robust imitation from demonstrations alone, without additional human corrections, in visual manipulation tasks.
Approach
Learn a latent world model, a discriminator-based reward model (moment-matching with gradient penalty) and a critic from hybrid expert/learner data, then at test time run MPPI over residual corrections to a Diffusion Policy's nominal plan, periodically distilling the search results back into the base policy (expert iteration).
Key results
- SAILOR significantly outperforms Diffusion Policy trained by BC across all 12 tasks and dataset scales considered.
- Scaling BC demonstrations by roughly 10x (to 200 demos) still does not match SAILOR's performance on the evaluated tasks.
- SAILOR is more interaction-efficient than DPPO-IRL, which does not match SAILOR even with 5x the interaction budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAILOR instead of a Diffusion Policy trained on the same demonstrations | Visual manipulation tasks across Robomimic, RoboSuite, and ManiSkill · 12 visual manipulation tasks | success rate | Helped |
| increase Diffusion Policy demonstration data by approximately 5–10× | Diffusion Policy behavioral cloning · Visual manipulation tasks | success rate: 5-10× | Mixed |
| increase Diffusion Policy training data to 200 demonstrations | Diffusion Policy behavioral cloning · Visual manipulation tasks | success rate: 200, ≈ 10× | Hurt |
| use SAILOR instead of DPPO-IRL | Imitation learning with a learned reward model · Visual manipulation tasks | success rate: 5× | Helped |
| increase MPPI planning samples up to 256 | SAILOR · Visual manipulation tasks | success rate: 256 | Helped |
| increase MPPI planning samples to eight times the training count | SAILOR · Visual manipulation tasks | success rate: 8× | No clear effect |
| add a warm-start phase using base-policy rollouts to train the WM, RM, and critic (ablation) | SAILOR · Visual manipulation tasks | success rate: ≈ 20% | Helped |
| increase the warm-start fraction from 0 to 0.1 (ablation) | SAILOR · Square, 50 demonstrations | performance: 0.1 | Helped |
| increase the warm-start fraction beyond 0.2 (ablation) | SAILOR · Square, 50 demonstrations | performance: 0.2 | No clear effect |
| train the world model on a hybrid of expert and learner data (ablation) | SAILOR · PokeCube; Cereal | performance | Helped |
| add expert iteration that distills search outputs into the base policy (ablation) | SAILOR · Visual manipulation tasks | performance | Helped |
| use explicit test-time residual planning instead of a residual policy (ablation) | SAILOR · Visual manipulation tasks | performance | Helped |
Surprising findings
- Performance did not degrade with 8x as many MPPI samples than used in training, indicating robustness to reward hacking.
- Explicit residual planning outperformed a learned residual policy trained with actor-critic, across the board.
Limitations
- Experiments use a per-task base diffusion policy and Dreamer world model rather than foundation VLAs or internet-scale world models.
- Requires an interaction budget of environment steps (up to 500k) and per-task training, so it is not zero-shot.
- Demonstrations are collected by a single operator on simulated tasks; real-world evaluation not reported.
Experimental setup
Datasets: Robomimic RoboSuite ManiSkill3
Benchmarks: Robomimic Lift Robomimic Can Robomimic Square RoboSuite Door RoboSuite Stack RoboSuite Bread RoboSuite Cereal RoboSuite Round RoboSuite BreadCan ManiSkill PullCube ManiSkill PokeCube ManiSkill LiftPeg
Metrics: success rate
Baselines: Diffusion Policy (behavioral cloning) DPPO-IRL Residual Policy
Builds on: Diffusion Policy DreamerV3 MPPI DPPO DAgger
Closest papers by meaning
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.90)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.90)
- Diffusion Model-Augmented Behavioral Cloning (ICML 2024 · similarity 0.89)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
- Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.89)
- When Does Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.