Research map / Multi-agent systems and games / Human-AI and goal-conditioned
OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OSVI-WM, which uses a latent world model and action model over ResNet-18 features of an expert video and agent frame to predict latent trajectories that are decoded into waypoints for one-shot visual imitation on unseen tasks.
Problem
One-shot visual imitation from a single expert video, where test tasks differ semantically or structurally from training tasks, and existing methods lack explicit environment dynamics models and generalize poorly to hard tasks.
Approach
Recursively predicts future latent agent states with a learned action model and world model conditioned on the expert demonstration latents, trained end-to-end in-domain with an auxiliary latent world-model L1 loss, and decodes the predicted latent trajectory into attributed 3D waypoints with a soft-DTW supervised MLP head and re-planning.
Key results
- On Meta-World, OSVI-WM achieves 83.8% average success (96% easy, 71.5% hard) versus 51.5% for AWDA with additional data.
- On Pick-and-Place, OSVI-WM reaches 100% success, matching AWDA.
- On real-world tasks, OSVI-WM reaches 96% on Two-Franka-PP, 92% on Human-Franka-PP (tied with AWDA), and 100% on Human-Franka-Push versus AWDA's 76%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OSVI-WM instead of AWDA with additional training data | Unseen-task simulation benchmarks · Meta-World Hard | success rate: 30 -> 71.5 | Helped |
| Add depth-based grasp correction to T-OSVI | Two-Franka-PP · Two-Franka-PP | overall success: 0 -> 4 | Helped |
| Add re-planning (ablation) | OSVI-WM · Meta-World Hard | success rate: ~42% -> 71.5% | Helped |
| Add world-model-guided trajectory generation with WM loss (ablation) | OSVI-WM · Meta-World | success rate | Helped |
| Add stop-gradient in WM loss (ablation) | OSVI-WM · Pick-and-Place | success rate: 20 -> 100 | Helped |
| Add stop-gradient in WM loss (ablation) | OSVI-WM · Meta-World | success rate: 76.5 -> 83.8 | Helped |
| Keep spatial pooling after the world model (ablation) | OSVI-WM · Pick-and-Place | success rate: 15 -> 100 | Helped |
| Keep spatial pooling after the world model (ablation) | OSVI-WM · Meta-World | success rate: 79.5 -> 83.8 | Helped |
| Use multi-step recursive rather than single-step trajectory prediction (ablation) | OSVI-WM · Pick-and-Place | success rate: 100 -> 100 | No clear effect |
| Use multi-step recursive rather than single-step trajectory prediction (ablation) | OSVI-WM · Meta-World | success rate: 67.3 -> 83.8 | Helped |
| Exclude all button-related tasks from training (ablation) | OSVI-WM · Meta-World Button-Press | success rate: 97% | No clear effect |
| Remove Window-Close from training (ablation) | OSVI-WM · Meta-World Window-Open | success rate: 95% -> 80% | Hurt |
Surprising findings
- Removing the stop-gradient in the world-model loss drops Pick-and-Place success from 100% to 20%, indicating model collapse.
- Excluding all button-related tasks from training still yields 97% on Button-Press, suggesting skill reuse from other training tasks.
Limitations
- Predicts only 3D position and gripper state, not end-effector orientation, limiting tasks like screwing or side-grasping.
- Focuses on single-task execution; sequential or multi-task settings not addressed.
- Performance depends on the train-test task division, and test tasks share the same manipulator family as training.
Experimental setup
Datasets: Meta-World Pick-and-Place Two-Franka-PP Human-Franka-PP Human-Franka-Push
Benchmarks: Meta-World Pick-and-Place Two-Franka-PP Human-Franka-PP Human-Franka-Push
Metrics: success rate
Baselines: DAML T-OSVI AWDA T-OSVI*
Builds on: AWDA T-OSVI DINO-WM DynaMo soft-DTW
Closest papers by meaning
- Deep Demonstration Tracing: Learning Generalizable Imitator Policy for Runtime Imitation from a Single Demonstration (ICML 2024 · similarity 0.90)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.89)
- What's the Move? Hybrid Imitation Learning via Salient Points (ICLR 2025 · similarity 0.88)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.88)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.