Research map / Video / Video-language understanding
VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an autoregressive video generation model (VQ tokenizer plus Llama-style transformer) with a Latent Dynamics Model on unlabeled Go and robot videos to learn rules, planning and control from pure visual input.
Problem
Whether a generative model can learn complex knowledge (game rules, reasoning, planning, robotic control) solely from unlabeled visual video, without text or action labels or RL.
Approach
Augment next-frame prediction with a Latent Dynamics Model that compresses multi-step future visual changes into a short sequence of quantized latent codes, which the autoregressive transformer predicts alongside frames; an inverse dynamics model maps generated frames/codes to actions.
Key results
- VideoWorld 300M reaches Elo 2317 on Video-GoBench, above KataGo-human-5d (2253), without search or RL.
- On CALVIN, VideoWorld (video-only) reaches 56.2/75.4/72.1% success on Push/Open-Close/Turn on-off, versus 17.3/24.1/19.2% for the basic transformer; with +30k data it reaches 72.7/91.0/93.8%.
- Ablation on Go: state-sequence training (idx 4) outperforms raw-video training (idx 5, Elo 1998 vs 2308 for the 300M transformer), showing compact visual-change representations matter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Latent Dynamics Model (LDM) to video prediction | VideoWorld on Video-GoBench · Video-GoBench test set | Tournament Elo: 1998±38 -> 2317±25 | Helped |
| use state sequences instead of video input | 300M transformer · Video-GoBench test set | Tournament Elo: 1998±38 -> 2308±21 | Helped |
| add 10k CALVIN trajectories | VideoWorld · CALVIN ABCD→D, Push | Task Success Rate (%): 56.2 -> 65.3 | Helped |
| add 30k CALVIN trajectories | VideoWorld · CALVIN ABCD→D, Push | Task Success Rate (%): 56.2 -> 72.7 | Helped |
| increase compression length from H=1 to H=5 (ablation) | VideoWorld · Go | Action-Value (%): 70.3 -> 73.9 | Helped |
| increase compression length from H=1 to H=5 (ablation) | VideoWorld · CALVIN, Push | Success rate: 33.7 -> 50.3 | Helped |
| replace the first latent code with a random token (ablation) | VideoWorld · Go | Act-Acc. (%): 80.9 -> 42.1 | Hurt |
| replace the second latent code with a random token (ablation) | VideoWorld · Go | Act-Acc. (%): 80.9 -> 76.6 | Hurt |
| replace the third latent code with a random token (ablation) | VideoWorld · Go | Act-Acc. (%): 80.9 -> 80.6 | No clear effect |
| increase LDM codebook size from 64,000 to 262,144 (ablation) | VideoWorld · Go | Act-Acc. (%): 80.9 -> 53.2 | Hurt |
| use original human games without KataGo re-annotation (ablation) | VideoWorld · Go | Act-Acc. (%): 75.7 -> 34.8 | Hurt |
Surprising findings
- Video-only training reaches legal rate near 100% on Go and learns rules such as ko, without explicit rule supervision.
- Compression length beyond 5 steps on Go causes LDM training to fail to converge, attributed to combinatorial board variation overwhelming the codebook.
Limitations
- Limited exploration of larger-scale models and data.
- Domain generalization limited; real-world deployment needs high-fidelity visual synthesis.
- Still early stage for learning knowledge from unlabeled videos; CALVIN performance still below action-supervised oracle without extra data.
Experimental setup
Datasets: Video-GoBench (KataGo self-play and OGS human games) CALVIN RLBench
Benchmarks: Video-GoBench CALVIN RLBench
Metrics: legal rate Elo rating action accuracy action-value ratio task success rate
Baselines: KataGo-human-1d KataGo-human-5d KataGo-human-9d MCIL HULC SPIL LCD Transformer (state-based, video-based)
Builds on: VQ-VAE MAGVIT-v2 Llama LAPO UniPi
Closest papers by meaning
- Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.89)
- Position: Video as the New Language for Real-World Decision Making (ICML 2024 · similarity 0.88)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
- SLVMBench: Skill Learning from Video Memory (NeurIPS 2026 · similarity 0.88)
- Vision Language Models are In-Context Value Learners (ICLR 2025 · similarity 0.87)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.