Research map / Reinforcement learning / Imitation learning
Reward-free World Models for Online Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IQ-MPC, a reward-free latent world model that combines inverse soft-Q learning with MPPI planning for online imitation learning from expert demonstrations.
Problem
Online imitation learning struggles on tasks with high-dimensional observations/actions and complex dynamics; adversarial reward-policy min-max optimization is unstable.
Approach
Learn a decoder-free latent dynamics model and derive rewards directly from the inverse soft-Q critic (r = T^pi Q), so no separate reward model is needed; plan with MPPI using a policy prior and decoded rewards, optimizing in Q-policy space instead of reward-policy space.
Key results
- On MyoSuite, IQ-MPC reaches success rates of 0.87 (Key Turn), 0.96 (Object Hold), and 0.73 (Pen Twirl), versus at most 0.72 for IQL+SAC and 0.55 for HyPE on Key Turn; other baselines score near 0 on Object Hold and Pen Twirl.
- On ManiSkill2 Pick Cube, IQ-MPC achieves 0.79 success vs 0.61 for IQL+SAC and 0.00 for CFIL+SAC.
- Recovered-reward Pearson correlation with ground truth is higher than IQL+SAC on all four tasks (e.g., Hopper Hop 0.88 vs 0.49, Walker Run 0.91 vs 0.65).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IQ-MPC instead of IQL+SAC | online imitation learning · MyoSuite Key Turn | success rate: 0.72$\pm$0.04 -> 0.87$\pm$0.03 | Helped |
| use IQ-MPC instead of IQL+SAC | online imitation learning · MyoSuite Object Hold | success rate: 0.00 $\pm$ 0.00 -> 0.96$\pm$0.03 | Helped |
| use IQ-MPC instead of IQL+SAC | online imitation learning · MyoSuite Pen Twirl | success rate: 0.00 $\pm$ 0.00 -> 0.73$\pm$0.05 | Helped |
| use IQ-MPC instead of IQL+SAC | online imitation learning · ManiSkill2 Pick Cube | success rate: 0.61$\pm$0.13 -> 0.79$\pm$0.05 | Helped |
| use IQ-MPC instead of IQL+SAC | online imitation learning · ManiSkill2 Lift Cube | success rate: 0.85 $\pm$ 0.04 -> 0.89$\pm$0.02 | Helped |
| use IQ-MPC instead of IQL+SAC for reward recovery | reward decoding · Hopper Hop | Pearson correlation: 0.49 -> 0.88 | Helped |
| use IQ-MPC instead of IQL+SAC for reward recovery | reward decoding · Cheetah Run | Pearson correlation: 0.79 -> 0.87 | Helped |
| use IQ-MPC instead of IQL+SAC for reward recovery | reward decoding · Walker Run | Pearson correlation: 0.65 -> 0.91 | Helped |
| use IQ-MPC instead of IQL+SAC for reward recovery | reward decoding · Quadruped Run | Pearson correlation: 0.88 -> 0.93 | Helped |
| add Wasserstein-1 gradient penalty (ablation) | IQ-MPC · ManiSkill2 Pick Cube | success rate: 0.51$\pm$0.11 -> 0.79$\pm$0.05 | Helped |
| add Wasserstein-1 gradient penalty (ablation) | IQ-MPC · DMControl locomotion tasks | performance | No clear effect |
| replace the value temporal-difference term with the initial-state value formulation (ablation) | IQ-MPC · Humanoid Walk | Q-estimation stability | Helped |
Surprising findings
- CFIL+SAC fails to train on Pen Twirl and gets near-zero success on several tasks, suggesting adversarial baselines are unstable in high-dimensional settings.
- Using only the policy prior without MPC accelerates training but may introduce greater instability.
Limitations
- Training is slower than model-free baselines due to MPC planning overhead.
- Designed for deterministic settings; robustness to noisy dynamics was only tested with small trembling probabilities (0.01-0.02).
- Gradient penalty helped only on manipulation tasks and gave no gain on locomotion, and some tasks showed instability with few expert trajectories (5 in Hopper Hop).
Experimental setup
Datasets: DMControl expert trajectories generated by TD-MPC2 MyoSuite expert trajectories generated by TD-MPC2 ManiSkill2 tasks
Benchmarks: DMControl MyoSuite ManiSkill2
Metrics: episode reward success rate Pearson correlation of recovered rewards
Baselines: IQL+SAC CFIL+SAC HyPE HyPER
Builds on: IQ-Learn TD-MPC2 TD-MPC SQIL MPPI
Closest papers by meaning
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.90)
- EvIL: Evolution Strategies for Generalisable Imitation Learning (ICML 2024 · similarity 0.90)
- Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model (ICLR 2025 · similarity 0.89)
- Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learning (AAAI 2025 · similarity 0.89)
- SEABO: A Simple Search-Based Method for Offline Imitation Learning (ICLR 2024 · similarity 0.89)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.