Research map / Video / Action recognition
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vista builds a generalizable driving world model by adapting Stable Video Diffusion with latent-replacement dynamic priors, dynamics and structure losses, and LoRA-based multi-modal action control, plus a self-uncertainty reward.
Problem
Existing driving world models generalize poorly to unseen environments, lose fine details at low frame rate and resolution, and support only limited action modalities.
Approach
Adapt SVD into a predictive driving model by injecting up to three clean condition frames via latent replacement, re-weighting the diffusion loss toward moving regions, supervising high-frequency structure in the Fourier domain, and adding unified multi-modal action conditioning via LoRA with action-independence training; the model's own conditional variance serves as an action reward.
Key results
- On nuScenes validation, Vista reaches FID 6.9 and FVD 89.4, versus 15.4 and 184.0 for the best prior driving world model GenAD (as reported in the paper's table).
- In human evaluation with 2640 answers from 33 participants, Vista is preferred over general-purpose video generators in over 70% of comparisons, per the abstract.
- Its uncertainty-based reward decreases as trajectory L2 error from ground truth increases on Waymo, a dataset unseen in training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GenAD with Vista | driving world model prediction on nuScenes validation · nuScenes validation | FID: 15.4 -> 6.9 | Helped |
| replace GenAD with Vista | driving world model prediction on nuScenes validation · nuScenes validation | FVD: 184.0 -> 89.4 | Helped |
| add a second dynamic prior instead of using one (ablation) | Vista action-free rollouts · nuScenes | Average Trajectory Difference: 3.785 -> 2.597 | Helped |
| add a third dynamic prior instead of using two (ablation) | Vista action-free rollouts · nuScenes | Average Trajectory Difference: 2.597 -> 1.820 | Helped |
| add a second dynamic prior instead of using one (ablation) | Vista action-free rollouts · Waymo | Average Trajectory Difference: 3.646 -> 2.901 | Helped |
| add a third dynamic prior instead of using two (ablation) | Vista action-free rollouts · Waymo | Average Trajectory Difference: 2.901 -> 2.052 | Helped |
| add a trajectory condition | Vista with 3 dynamic priors · nuScenes | Average Trajectory Difference: 1.820 -> 0.835 | Helped |
| add an angle and speed condition | Vista with 3 dynamic priors · nuScenes | Average Trajectory Difference: 1.820 -> 0.832 | Helped |
| add a trajectory condition | Vista with 3 dynamic priors · Waymo | Average Trajectory Difference: 2.052 -> 1.140 | Helped |
| add the action independence constraint (ablation) | Vista trajectory control learning · Subset FVD on nuScenes | FVD: 138.8 -> 130.7; 232.9 -> 230.8; 368.2 -> 345.7; 132.3 -> 118.9 | Helped |
| add the dynamics enhancement loss (ablation) | Vista high-fidelity prediction · Driving video prediction | motion realism | Helped |
| add the structure preservation loss (ablation) | Vista high-fidelity prediction · Driving video prediction | structural detail | Helped |
Surprising findings
- Plain L2 error fails to judge some actions that the uncertainty-based reward can assess, per the case study.
- Replacing latents for condition frames requires no extra channels and does not degrade SVD's generation quality when applied directly.
Limitations
- Computationally expensive at high spatiotemporal resolution, especially in downstream use.
- Prediction may degrade in long-horizon rollouts or under drastic view shifts.
- Action controls can fail for ambiguous intentions such as commands and goal points.
Experimental setup
Datasets: OpenDV-YouTube nuScenes Waymo CODA
Benchmarks: nuScenes Waymo CODA OpenDV-YouTube-val
Metrics: FID FVD human preference (two-alternative forced choice) trajectory difference via inverse dynamics model L2 correlation of reward
Baselines: DriveGAN DriveDreamer WoVoGen Drive-WM GenAD Stable Video Diffusion
Builds on: Stable Video Diffusion EDM LoRA GenAD Drive-WM
Closest papers by meaning
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.88)
- DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model (NeurIPS 2024 · similarity 0.88)
- ReSim: Reliable World Simulation for Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.88)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.87)
- UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.