Research map / Video / Video generation
Position: Video as the New Language for Real-World Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing video generation can play the role for the physical world that language modeling plays for the digital world, illustrated with demonstrations across vision tasks, games, robotics, self-driving, and science.
Problem
Video generation has been used mostly for entertainment, while video data captures physical-world information that is hard to express in language; the paper asks how video generation can serve as a unified representation, interface, and decision-making tool for real-world tasks.
Approach
Frames video as a unified representation and task interface analogous to text for LLMs, arguing video generation can act as planner, agent, compute engine, and environment simulator via in-context learning, planning, and RL, supported by illustrative generated examples.
Key results
- Proof-of-concept Minecraft autoregressive model generates action-observation trajectories serving as both world model and policy, with no quantitative metrics reported.
- A 10.7B masked latent-action model is shown to produce consistent control variables across visual prompts (qualitative).
- Single video generation model trained on Open X-Embodiment produces realistic robot video plans (qualitative).
Surprising findings
- Generated videos of a robot hand given an unrealistic instruction ('wash hand') produce egocentric motions to realistically approximate the task rather than following it literally.
- Video generation and world-model/policy roles can be combined in one autoregressive model, though latent state space models may be preferable for planning.
Limitations
- Examples are qualitative and illustrative with no quantitative evaluation.
- Hallucination: objects randomly appear or disappear and implausible dynamics occur.
- Limited data coverage and labels for domain-specific video; limited generalization to high-resolution or unfamiliar domains.
Experimental setup
Datasets: VPT contractor data (Minecraft) Open X-Embodiment Language Table MineDojo internet video
Builds on: Dreamer MuZero VPT MaskGIT Open X-Embodiment
Closest papers by meaning
- Video Language Planning (ICLR 2024 · similarity 0.89)
- VideoWorld: Exploring Knowledge Learning from Unlabeled Videos (CVPR 2025 · similarity 0.88)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.87)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.87)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.