Research map / Video / Video segmentation and tracking
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Drive-WM is a multiview latent video diffusion world model for driving that generates consistent multiview videos conditioned on unified inputs and uses it for image-reward-based tree planning with VAD.
Problem
Autonomous driving planners trained on expert trajectories generalize poorly to out-of-distribution cases; existing driving world models handle only single-view video or multiview images, and no world model had been applied to end-to-end planning.
Approach
Extend a pretrained Stable Diffusion image model with temporal and multiview attention layers, factorize multiview generation into reference and stitched views for consistency, use a unified condition interface (images, text, layouts, actions), and plan by rolling out a tree of imagined futures scored with image-based perception rewards.
Key results
- Multiview video generation reaches 15.8 FID and 122.7 FVD on nuScenes, versus 73.4 FID / 502.3 FVD for DriveGAN and 52.6 / 452.0 for DriveDreamer.
- Factorized multiview generation raises the KPM multiview consistency score from 45.8% to 94.4%.
- Tree-based planning with the world model gives average L2 0.80 m and collision 0.26%, close to VAD with ground-truth commands (0.72 m, 0.22%) and better than VAD with sampled commands (0.87 m, 0.47%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tree-based planning to select among three sampled driving commands | VAD planner with Drive-WM · nuScenes val planning | L2 (m), Avg.: 0.87 -> 0.80 | Helped |
| Use tree-based planning to select among three sampled driving commands | VAD planner with Drive-WM · nuScenes val planning | Collision (%), Avg.: 0.47 -> 0.26 | Helped |
| Add layout condition (ablation) | Drive-WM · nuScenes val multiview video generation | FID: 18.9 -> 15.8 | Helped |
| Add layout condition (ablation) | Drive-WM · nuScenes val multiview video generation | KPM (%): 44.6 -> 45.8 | Helped |
| Add temporal embedding (ablation) | Drive-WM · nuScenes val multiview video generation | FID: 20.3 -> 18.9 | Helped |
| Add temporal layers (ablation) | Drive-WM multiview temporal tuning · nuScenes val multiview video generation | FID: 23.3 -> 16.2 | Helped |
| Add view layers (ablation) | Drive-WM multiview temporal tuning · nuScenes val multiview video generation | KPM (%): 40.8 -> 40.9 | No clear effect |
| Use factorized generation instead of joint modeling (ablation) | Drive-WM · nuScenes val multiview video generation | KPM (%): 45.8 -> 94.4 | Helped |
| Use factorized generation instead of joint modeling (ablation) | Drive-WM · nuScenes val multiview video generation | FID: 15.8 -> 16.4 | Hurt |
| Use factorized generation instead of joint modeling (ablation) | Drive-WM · nuScenes val multiview video generation | FVD: 122.7 -> 116.6 | Helped |
| Add map reward (ablation) | Drive-WM tree-based planner · nuScenes val planning | Collision (%), Avg.: 0.93 -> 0.39 | Helped |
| Add object reward (ablation) | Drive-WM tree-based planner · nuScenes val planning | Collision (%), Avg.: 0.93 -> 0.27 | Helped |
Surprising findings
- Under a 0.5 m lateral ego shift, VAD's average collision rate rises from 0.22% to 1.59%, and fine-tuning on world-model-generated recovery data reduces it to 0.91%.
Limitations
- Generation is limited to low resolution (384x192) cropped from nuScenes images.
- Planning evaluation relies on a single planner (VAD) and a single dataset (nuScenes).
- Non-vectorized rewards via GPT-4V are only illustrated qualitatively in the appendix.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: FID FVD mAP_obj mAP_map mIoU KPM L2 collision rate
Baselines: BEVGen BEVControl MagicDrive DriveGAN DriveDreamer LayoutDiffusion GLIGEN VAD
Builds on: Stable Diffusion VideoLDM VAD MagicDrive
Closest papers by meaning
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.90)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.89)
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation (AAAI 2025 · similarity 0.89)
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction (NeurIPS 2025 · similarity 0.89)
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving (NeurIPS 2025 · similarity 0.89)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.