Research map / Reinforcement learning / Imitation learning
Robust Visual Imitation Learning with Inverse Dynamics Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RILIR, a visual imitation learning method that learns an inverse-dynamics state representation jointly with a TD3 critic and combines optimal-transport trajectory-matching rewards with a GAIL-style discriminator reward, evaluated under visual perturbations on DMC and Meta-World.
Problem
Imitation learning from high-dimensional image observations fails when the learning environment visually differs from the expert demonstration environment (e.g., distractors, noise, random masking).
Approach
Learn an action-related state embedding via an inverse dynamics objective over both expert and behavior transitions (jointly with the critic), and build a compound reward from Sinkhorn-based Wasserstein trajectory matching plus a discriminator reward computed in that embedding space.
Key results
- Across visual shift settings, RILIR significantly outperforms BC, DAC, ROT, DA-ROT, PatchAIL and SeMAIL on DMC tasks, reaching near-expert performance in most environments.
- In manipulation and locomotion tasks with random masking, the return or success gap to the expert is reported as less than 5%.
- Ablations show removing the discriminator reward mainly slows learning, while removing the inverse dynamics representation mainly hurts convergent performance, especially on Drawer Close.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inverse-dynamics representation learning | RILIR · Visual control tasks with visual shifts | average return | Helped |
| Combine trajectory-matching and discriminator rewards | RILIR · Visual control tasks with visual shifts | average return | Helped |
| Use RILIR instead of baseline methods | RILIR · Clean environments | average return | No clear effect |
| Apply DrQ data augmentation to ROT | ROT · Visual control tasks with visual shifts | average return | Mixed |
| Use patch-based image rewards | PatchAIL · Noisy visual environments | average return | Hurt |
| Use separated forward models | SeMAIL · Walker Stand with changing backgrounds | average return | No clear effect |
| Use separated forward models | SeMAIL · Environments with easier visual perturbations | learning efficiency | Hurt |
| Remove inverse-dynamics representation learning (ablation) | RILIR · Hammer and Drawer Close with random masking | convergent performance | Hurt |
| Add inverse-dynamics representation learning (ablation) | RILIR · Hammer and Drawer Close with random masking | convergent performance | Helped |
| Remove discriminator rewards, retaining only trajectory-matching rewards (ablation) | RILIR · Hammer and Drawer Close with random masking | learning efficiency | Hurt |
| Add discriminator rewards to trajectory-matching rewards (ablation) | RILIR · Hammer and Drawer Close with random masking | learning efficiency | Helped |
| Use cosine rather than Euclidean distance for the transport cost (ablation) | RILIR · Appendix D ablation (task unspecified) | performance | Helped |
Surprising findings
- PatchAIL performs similarly to DAC in clean environments but degrades severely under visual noise, suggesting patch-averaged rewards do not help with visual shifts.
Limitations
- May not work well when learning-environment observations differ greatly from expert demonstrations.
- Inverse dynamics objective assumes invariant dynamics; fails when dynamics differ between environments.
Experimental setup
Datasets: DeepMind Control Suite expert demonstrations Meta-World expert demonstrations Distracting Control Suite (easy) environments
Benchmarks: DeepMind Control Suite (CartPole Swingup, Walker Stand, Finger Spin) Meta-World (Hammer, Drawer Close) Distracting Control Suite easy
Metrics: average episode return success rate
Baselines: BC DAC ROT DA-ROT PatchAIL SeMAIL
Builds on: TD3 GAIL ROT SeMAIL PatchAIL
Closest papers by meaning
- State Alignment-based Imitation Learning (ICLR 2020 · similarity 0.90)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.90)
- Imitation from Observations with Trajectory-Level Generative Embeddings (NeurIPS 2026 · similarity 0.89)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.88)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.88)
- Causal Imitation for Markov Decision Processes: a Partial Identification Approach (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.