Research map / Vision-language and multimodal / Robot action and navigation
VideoVLA: Video Generators Can Be Generalizable Robot Manipulators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
VideoVLA adapts the pre-trained CogVideoX-5B video diffusion transformer into a joint video-and-action generator that predicts action chunks and future frame latents, improving generalization to novel objects and cross-embodiment skills on SIMPLER and a real Realman robot.
Problem
Vision-language-action models built on pre-trained vision-language understanding backbones have limited generalization to novel tasks, objects, and settings in robot manipulation.
Approach
Initialize a multimodal Diffusion Transformer from a large pre-trained video generator (CogVideoX-5B), treat 7-D actions as an additional modality, and jointly denoise future video latents and actions so that imagined futures and executed actions stay consistent.
Key results
- Average success of 63.0% across 12 SIMPLER tasks, the highest overall versus CogACT (62.6%) and pi0 (50.0%).
- Novel-object pick-up in SIMPLER averages 65.2% versus 42.4% for CogACT; new-skill transfer averages 48.6% versus 20.4% for CogACT.
- Real-world task average 64.6% versus 58.4% for CogACT; removing the video loss drops SIMPLER in-domain average from 80.4% to 27.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoVLA instead of the strongest baseline | SIMPLER Google new skills transfer · SIMPLER Google new skills | success rate (Avg.): 20.4 -> 48.6 | Helped |
| Use VideoVLA instead of the strongest baseline | Realman novel-object manipulation · Realman novel objects | success rate (Avg.): 26.9 -> 50.6 | Helped |
| Use VideoVLA instead of the strongest baseline | Realman cross-embodiment skill transfer · Realman cross-embodiment new skills | success rate (Avg.): 35.1 -> 58.0 | Helped |
| Replace CogVideoX-5B with OpenSora-1.1 (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 50.2 | Hurt |
| Remove large-scale video pre-training (ablation) | VideoVLA with CogVideoX-5B · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 12.6 | Hurt |
| Reduce the predicted future horizon from 49 to 25 frames (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 77.4 | Hurt |
| Reduce the predicted future horizon from 49 to 13 frames (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 75.2 | Hurt |
| Remove the video denoising loss while retaining joint video-action modeling (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 27.0 | Hurt |
| Remove future-video prediction and predict actions only (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 25.5 | Hurt |
| Replace bidirectional attention with causal masking (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 75.5 | Hurt |
| Use asynchronous training noising with synchronous inference instead of a synchronous schedule (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 73.8 | Hurt |
| Use asynchronous training and asynchronous inference instead of a synchronous schedule (ablation) | VideoVLA · SIMPLER Google VM tasks | success rate (Avg.): 80.4 -> 71.0 | Hurt |
Surprising findings
- Training from scratch without video pre-training drops the in-domain average from 80.4% to 12.6%, showing the video-generator initialization is critical.
- Human-judged imagined futures succeed more often (84.0% novel objects, 63.4% new skills) than actual executions (65.2%, 48.6%).
Limitations
- Slow inference (about 1.1 s per action chunk, roughly 3 Hz control) due to the large video generator.
- Reliance on a 5B-parameter general-purpose video model rather than a robot-specific one.
- Imagined-future success is judged by human evaluation only, not automatically.
Experimental setup
Datasets: Open X-Embodiment SIMPLER YCB GSO self-collected Realman teleoperation dataset
Benchmarks: SIMPLER (Visual Matching) SIMPLER (Variant Aggregation) YCB/GSO novel objects in SIMPLER real-world Realman tasks
Metrics: success rate motion similarity (SIFT + SAM-PT + Hungarian matching)
Baselines: OpenVLA SpatialVLA pi0 CogACT Octo RT-1-X RT-2-X UVA VPP
Builds on: CogVideoX DiT DDPM Open X-Embodiment T5
Closest papers by meaning
- DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.92)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.91)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.91)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.90)
- VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models (NeurIPS 2025 · similarity 0.90)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.