Research map / Video / Video generation
VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
VidMan pre-trains a video diffusion transformer (Open-Sora STDiT) on Open X-Embodiment to predict future frames, then adapts it into an implicit inverse-dynamics action model via layer-wise self-attention adapters and a diffusion action head, evaluated on CALVIN and OXE offline.
Problem
Robot manipulation action prediction is limited by scarce robot data; directly fitting actions from observations ignores the relation between visual dynamics and actions, leading to poor data utilization.
Approach
A two-stage dual-process-inspired training: first pre-train a video diffusion transformer to predict future robot frames (dynamics awareness), then reuse its parameters with a layer-wise self-attention adapter and a diffusion action head to predict actions without generating frames (implicit inverse dynamics).
Key results
- On CALVIN (zero-shot, train A/B/C test D), VidMan reaches 3.42 average length versus GR-1's 3.06 (about 11.7% relative improvement).
- On OXE offline evaluation, VidMan's avg xyz/angle accuracy on Cable Routing is 24.8 vs Octo-base 14.9, and on Autolab UR5 is 33.2 vs 24.3.
- Ablation: co-training frame generation with action in stage 2 drops CALVIN average length from 3.42 to 2.70; removing the layer-wise adapter drops it to 1.54.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VidMan instead of GR-1 | CALVIN zero-shot evaluation · CALVIN | Avg. Len.: 3.06 -> 3.42 | Helped |
| pretrain on OXE instead of no video pretraining (ablation) | VidMan · CALVIN | Avg. Len.: 2.89 -> 3.42 | Helped |
| pretrain on Ego4d instead of no video pretraining (ablation) | VidMan · CALVIN | Avg. Len.: 2.89 -> 3.29 | Hurt |
| co-train video generation and action prediction in stage 2 instead of action-only supervision (ablation) | VidMan · CALVIN | Avg. Len.: 3.42 -> 2.70 | Hurt |
| add the layer-wise adapter (ablation) | VidMan · CALVIN | Avg. Len.: 1.54 -> 3.42 | Helped |
| train Open-Sora blocks as well as the adapter instead of freezing the blocks (ablation) | VidMan · CALVIN | Avg. Len.: 2.98 -> 3.42 | Helped |
| increase frame sampling interval from 1 to 3 (ablation) | VidMan · Bridge | MSE: 2.80 -> 0.89 | Helped |
| increase frame sampling interval from 3 to 4 (ablation) | VidMan · CALVIN | Avg. Len.: 3.42 -> 3.03 | Hurt |
| use pure noise instead of zero embeddings in stage 2 (ablation) | VidMan · Bridge | MSE: 12.1 -> 4.8 | Helped |
| use two historical frames instead of one (ablation) | VidMan · Bridge | Avg. Len.: 2.80 -> 3.42 | Helped |
| increase future frames from one to two (ablation) | VidMan · Bridge | Avg. Len.: 2.80 -> 3.42 | Helped |
Surprising findings
- Jointly training frame generation and action prediction in the second stage severely degrades performance (3.42 to 2.70), approaching baseline levels.
- Pretraining on general web video (Ego4D) slightly hurt CALVIN performance relative to no pretraining in the stated comparison (3.29 vs 2.89 for no pretraining, so it actually helped; robot-domain OXE pretraining helped most).
Limitations
- Operates only on 2D third-person and wrist images without 3D perception.
- Limited capacity for complex human language instructions (CLIP text encoder).
- Perception is coarse, processing the whole image as a single input without fine-grained object cues.
Experimental setup
Datasets: Open X-Embodiment (25 selected datasets) CALVIN Bridge Taco Play Cable Routing Autolab UR5 Ego4D RLBench
Benchmarks: CALVIN Bridge Taco Play Cable Routing Autolab UR5 RLBench
Metrics: average task length success rate per chain position MSE xyz accuracy angle accuracy FID FVD
Baselines: GR-1 RoboFlamingo SuSIE HULC MCIL RT-1 3D Diffuser Actor 3D Diffusion Policy RT-1-X Octo-small Octo-base VidMan-GPT
Builds on: Open-Sora GR-1 Octo diffusion policy
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.91)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.91)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.90)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.89)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.89)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.