Research map / Video / Video segmentation and tracking
Overcoming Challenges of Long-Horizon Prediction in Driving World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Orbis, a 469M-parameter continuous flow-matching driving world model trained on 280h of front-camera video, and uses a hybrid tokenizer to show continuous autoregressive modeling beats discrete MaskGIT-style token modeling for long-horizon driving prediction.
Problem
Driving world models produce high-quality short clips but fail on long-horizon generation, turning maneuvers, and realistic ego-motion; it is unclear whether discrete tokens or continuous latents are responsible.
Approach
A hybrid semantic+detail tokenizer (DINOv2-distilled factorized tokens) that can be used either quantized or continuous, enabling a controlled side-by-side comparison of masked generative (discrete) and flow-matching (continuous) next-frame autoregressive world models trained only on raw video.
Key results
- Orbis (flow matching, 469M) achieves FVD 134.06 on nuPlan, 167.57 on Waymo, and 239.20 on nuPlan-turns over 6s rollouts, beating Vista, GEM, DrivingWorld and Cosmos on FVD and on JEDi for nuPlan and nuPlan-turns.
- Orbis-FM ESTIMATES: consistently outperforms the discrete Orbis-MG across DiT, STDiT and CDiT architectures and 12/30 sampling steps in the small-scale ablation (e.g., ST: 246.0 vs 571.7 FVD at 30 steps).
- Orbis trajectory precision on Fréchet is 0.47 vs 0.39 (Vista) and 0.33 (GEM) on turning events.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use continuous flow matching instead of discrete masked generation (ablation) | Orbis world model · BDD100K, 60-frame generation | FVD: 533.3 -> 360.9 (12 steps); 571.7 -> 246.0 (30 steps) | Helped |
| add DINO distillation to the tokenizer without token factorization (ablation) | Orbis-MG · BDD100K, 60-frame generation | FVD: 1331.28 -> 1214.34 | Helped |
| add token factorization to the DINO-distilled tokenizer (ablation) | Orbis-MG · BDD100K, 60-frame generation | FVD: 1214.34 -> 533.28 | Helped |
| add DINO distillation without token factorization (ablation) | Orbis tokenizer · BDD100K tokenizer reconstruction | rFID: 9.33 -> 12.17 | Hurt |
| add token factorization with DINO distillation (ablation) | Orbis tokenizer · BDD100K tokenizer reconstruction | rFID: 12.17 -> 9.10 | Helped |
| add DINO distillation and token factorization (ablation) | Orbis continuous model · BDD100K, 60-frame generation | FVD: 240.34 -> 246.11 | No clear effect |
| condition generation on ego-motion (ablation) | Orbis · nuPlan validation, 5s trajectories | ADE: 5.20 -> 2.40 | Helped |
| replace DiT with STDiT (ablation) | Orbis-FM · BDD100K, 120-frame generation | FVD: 287.03 -> 273.69 | Helped |
| replace ST architecture with DiT (ablation) | Orbis-MG · BDD100K, 60-frame generation, 12 steps | FVD: 533.3 -> 769.0 | Hurt |
| replace ST architecture with CDiT (ablation) | Orbis-MG · BDD100K, 60-frame generation, 12 steps | FVD: 533.3 -> 1552.3 | Hurt |
| replace ST architecture with DiT (ablation) | Orbis-FM · BDD100K, 60-frame generation, 30 steps | FVD: 246.0 -> 274.2 | Hurt |
| replace ST architecture with CDiT (ablation) | Orbis-FM · BDD100K, 60-frame generation, 30 steps | FVD: 246.0 -> 246.1 | No clear effect |
Surprising findings
- Factorized tokens with DINO distillation are essential for the discrete model, whereas the continuous model is robust to tokenizer design changes.
- The discrete model's classifier copies the last context token about 45% of the time (vs 29% for encoded frames), causing early rollout termination.
Limitations
- Could not determine why much larger public video diffusion models (SVD-derived) fail on long rollouts.
- Detailed content such as traffic lights and signs not generated reliably; traffic actors do not always follow rules.
- No error bars or multiple seeds; not evaluated on downstream planning or decision making.
Experimental setup
Datasets: BDD100K OpenDV Honda HAD Honda HDD ONCE NuScenes NuPlan Waymo Open Dataset
Benchmarks: nuPlan nuPlan-turns Waymo
Metrics: FVD JEDi FID Frechet precision/recall ADE precision/recall PSNR SSIM DOVER
Baselines: Vista GEM DrivingWorld Cosmos
Builds on: VQGAN MaskGIT DiT EQ-VAE GAIA-1
Closest papers by meaning
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.87)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.87)
- Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR 2024 · similarity 0.87)
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024 · similarity 0.86)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.86)
- DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.