Research map / Video / Video generation
SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMPO, a hybrid scale-wise autoregressive world model that combines next-scale VAR-style intra-frame generation with causal inter-frame prediction, plus an asymmetric multi-scale VQ tokenizer and trajectory-aware motion prompts from CoTracker3.
Problem
Autoregressive world models for action-conditioned video prediction suffer from broken spatial structure from raster-scan flattening, slow token-by-token decoding with error accumulation, and weak modeling of salient robot/object motion.
Approach
Replace raster-scan token generation within each frame with coarse-to-fine next-scale prediction (VAR) while keeping causal autoregression across frames; use dense tokens for observed frames and sparse coarse tokens for future frames; inject point-tracking-based trajectory prompts from observed frames.
Key results
- On action-conditioned RoboNet at 256x256, SAMPO reports FVD 57.1 at 64x64 and 175.3 FVD for SAMPO-L at 256x256 versus 197.9 for iVideoGPT.
- On VP2 visual planning, SAMPO averages 72.2% success versus 70.1% for iVideoGPT.
- SAMPO-S with spatial scales [1,2,3,4,5,6] runs at 2.06 s/video versus 9.05 s/video for the AR baseline, the basis of the claimed 4.4x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace raster-scan AR with hybrid scale-wise autoregression (ablation) | SAMPO-S · RoboNet action-conditioned 256×256 | FVD: 197.9 -> 227.4 | Hurt |
| Add motion prompt (ablation) | SAMPO-S · RoboNet action-conditioned 256×256 | FVD: 227.4 -> 217.8 | Helped |
| Add motion prompt (ablation) | SAMPO-S · RoboNet action-conditioned 256×256 | SSIM: 76.4 -> 78.8 | Helped |
| Add temporal position embedding alongside motion prompt (ablation) | SAMPO-S · RoboNet action-conditioned 256×256 | FVD: 217.8 -> 193.8 | Helped |
| Add temporal position embedding alongside motion prompt (ablation) | SAMPO-S · RoboNet action-conditioned 256×256 | SSIM: 78.8 -> 81.5 | Helped |
| Scale the model from SAMPO-S to SAMPO-B (ablation) | SAMPO-S · RoboNet action-conditioned 256×256 | FVD: 193.8 -> 184.1 | Helped |
| Scale the model from SAMPO-B to SAMPO-L (ablation) | SAMPO-B · RoboNet action-conditioned 256×256 | FVD: 184.1 -> 175.3 | Helped |
| Use spatial scales [1, 2, 3, 4, 5, 6] instead of [1, 2, 4, 8] (ablation) | SAMPO-S · BAIR | FVD: 80.7 -> 55.5 | Helped |
| Use spatial scales [1, 2, 3, 4, 5, 6] instead of [1, 2, 4, 8] (ablation) | SAMPO-S · BAIR | inference time: 1.61 s / vid -> 2.06 s / vid. | Hurt |
| Use spatial scales [1, 2, 4, 8, 10] instead of [1, 2, 4, 8] (ablation) | SAMPO-S · BAIR | FVD: 80.7 -> 73.1 | Helped |
| Use spatial scales [1, 2, 4, 8, 10] instead of [1, 2, 4, 8] (ablation) | SAMPO-S · BAIR | inference time: 1.61 s / vid -> 3.27 s / vid | Hurt |
| Use SAMPO instead of iVideoGPT | Video prediction model · RoboNet action-conditioned 256×256 | FVD: 197.9 -> 175.3 | Helped |
Surprising findings
- Smaller inter-scale strides (more scales with gradual resolution steps) gave better FVD than fewer, larger scale jumps under similar compute.
Limitations
- Motion prompts require at least two observed frames, so single-frame inputs are unsupported.
- Long-horizon rollouts accumulate errors over time.
- Motion prompt is an external module rather than learned end-to-end.
Experimental setup
Datasets: Open X-Embodiment BAIR Robot Pushing RoboNet 1X World Model VP2 (Robosuite, RoboDesk) Meta-World
Benchmarks: BAIR Robot Pushing RoboNet 1X World Model VP2 Meta-World
Metrics: FVD PSNR SSIM LPIPS success rate inference time
Baselines: iVideoGPT MaskViT FitVid MCVD SVG GHVAE MAGVIT DreamerV3 Struct-VRNN
Builds on: VAR iVideoGPT VQGAN CoTracker3 DrQ-v2
Closest papers by meaning
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.89)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.88)
- World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories (NeurIPS 2026 · similarity 0.88)
- Video World Models with Long-term Spatial Memory (NeurIPS 2025 · similarity 0.88)
- Learning Visual Feature-Based World Models via Residual Latent Action (NeurIPS 2026 · similarity 0.88)
- DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.