Research map / Video / Video segmentation and tracking
iVideoGPT: Interactive VideoGPTs are Scalable World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iVideoGPT, a scalable autoregressive transformer world model with compressive conditional VQGAN tokenization, pre-trained on ~1.4M robot and human manipulation trajectories and adapted to video prediction, visual planning, and visual MBRL.
Problem
Existing video generation models lack step-level action interactivity, while recurrent world models for RL lack scalability to complex, in-the-wild data; the paper asks how to combine interactivity and scalability for visual world models.
Approach
Tokenize future frames with a conditional VQGAN whose prediction encoder/decoder cross-attends to context-frame features, yielding 4x4 tokens per future frame (16x fewer than 16x16 per-frame tokens), and feed frames, actions, and rewards as one token sequence to a LLaMA-style GPT-like transformer trained with next-token prediction.
Key results
- Action-conditioned iVideoGPT reaches FVD 60.8 on BAIR 64x64 (vs MaskViT 70.5), and FVD 197.9 at 256x256 on RoboNet (vs MaskViT 211.7).
- On VP^2 RoboDesk, iVideoGPT outperforms all baselines in two tasks by a large margin and is comparable on average to SVG' (per paper).
- Model-based MBPO with iVideoGPT is the first reported successful MBPO on visual continuous control and matches or exceeds DreamerV3 on Meta-World six tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition the model on actions instead of using action-free prediction | iVideoGPT · BAIR, 64×64 | FVD: 75.0±0.20 -> 60.8±0.08 | Helped |
| use iVideoGPT instead of FitVid | action-conditioned video prediction · RoboNet, 64×64 | FVD: 62.5 -> 63.2±0.01 | Hurt |
| use iVideoGPT instead of FitVid | action-conditioned video prediction · RoboNet, 64×64 | PSNR: 28.2 -> 27.8±0.01 | Hurt |
| use iVideoGPT instead of FitVid | action-conditioned video prediction · RoboNet, 64×64 | SSIM: 89.3 -> 90.6±0.02 | Helped |
| use iVideoGPT instead of FitVid | action-conditioned video prediction · RoboNet, 64×64 | LPIPS: 2.4 -> 4.9±0.00 | Hurt |
| use iVideoGPT instead of MaskViT | action-conditioned video prediction · RoboNet, 256×256 | FVD: 211.7 -> 197.9±0.66 | Helped |
| use iVideoGPT instead of MaskViT | action-conditioned video prediction · RoboNet, 256×256 | LPIPS: 17.0 -> 14.7±0.01 | Helped |
| pre-train on mixed manipulation videos before fine-tuning | iVideoGPT · BAIR with 100 or 1,000 action-free trajectories | video prediction performance | Helped |
| pre-train before fine-tuning | iVideoGPT · BAIR with full downstream data | video prediction performance | No clear effect |
| increase the transformer from 138M to 436M parameters | iVideoGPT · pre-training dataset | validation loss | Helped |
| use conditional 4×4 future-frame tokenization rather than a standard 16×16 tokenizer | iVideoGPT tokenizer · RoboNet video prediction | computational efficiency: 16× | Helped |
| use conditional 4×4 future-frame tokenization rather than a standard 16×16 tokenizer | iVideoGPT tokenizer · RoboNet video prediction | reconstruction quality | Hurt |
Surprising findings
- Pre-training DreamerV3-XL on the same data yields low-quality blurred predictions, suggesting recurrent world models do not benefit from large-scale pre-training in this setting.
- Pre-trained transformer zero-shot predicts natural gripper motion on unseen BAIR robot; adapting only the tokenizer gives correct gripper type.
Limitations
- Compressive tokenization assumes initial frames provide sufficient context, which may fail for long videos and significant camera motion.
- Limited diversity of public robot data; zero-shot generalization to unseen robots is limited.
- Evaluated only in visually simple simulation; benefits of larger model scale not observed on real-robot tasks.
Experimental setup
Datasets: Open X-Embodiment Something-Something-v2 BAIR RoboNet VP^2 Meta-World
Benchmarks: BAIR RoboNet VP^2 Meta-World
Metrics: FVD PSNR SSIM LPIPS success rate interquartile mean
Baselines: MaskViT MAGVIT FitVid MCVD SVG DreamerV3 VideoGPT SVG'
Builds on: VQGAN VideoGPT MBPO DrQ-v2 ContextWM
Closest papers by meaning
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.86)
- Learning Visual Feature-Based World Models via Residual Latent Action (NeurIPS 2026 · similarity 0.86)
- SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models (NeurIPS 2025 · similarity 0.86)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.86)
- Exploration-Driven Generative Interactive Environments (CVPR 2025 · similarity 0.86)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.