Research map / Video / Video generation
DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DriveScape is an end-to-end multi-view driving video generator built on Stable Video Diffusion that fuses neighboring-view and keyframe features via a Bi-Directional Modulated Transformer (BiMoT) to enable high-resolution, sparse-condition controllable generation on nuScenes.
Problem
Generating high-resolution multi-view driving videos with 3D condition control is memory- and compute-expensive when all views are denoised jointly, and nuScenes annotations at 2Hz make sparse-condition, high-frame-rate generation hard.
Approach
Generate key views and neighbor views in sequence (rather than denoising all views jointly), fuse neighbor-camera videos and keyframe conditions into the latent space, and inject sparse maps/3D boxes through a BiMoT module with two opposite-direction cross-attentions plus temporal self-attention.
Key results
- Achieves FID 8.34 and FVD 76.39 on nuScenes validation, versus 11.15/116.00 for reproduced SparseCtrl and 15.80/122.70 for Drive-WM.
- Ablation: removing the BiMoT module raises FID from 8.34 to 20.98 and FVD from 76.39 to 142.81.
- Generates 576x1024 video at 10 fps under sparse (2Hz) conditions; downstream StreamPETR NDS drops from 53.7 (oracle) to 44.5 with generated data, vs 32.1 for Panacea.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DriveScape instead of SparseCtrl | Multi-view video generation on nuScenes · nuScenes validation | FID: 11.15 -> 8.34 | Helped |
| Use DriveScape instead of Drive-WM | Video generation on nuScenes · nuScenes validation | FVD: 122.70 -> 76.39 | Helped |
| Use DriveScape instead of DrivingDiffusion | Driving video generation · nuScenes validation | Road mIoU: 63.20 (-10.48) -> 68.21 (-5.47) | Helped |
| Use DriveScape instead of DrivingDiffusion | Driving video generation · nuScenes validation | Vehicle mIoU: 31.60 (-3.24) -> 30.27 (-4.57) | Hurt |
| Use DriveScape instead of Delphi | Driving video generation · nuScenes validation | Object NDS: 36.58 (-4.63) -> 36.5 (-4.71) | No clear effect |
| Use DriveScape instead of Panacea | Video generation evaluated with StreamPETR · nuScenes validation | Object NDS: 32.1 (-14.8) -> 44.5 (-9.2) | Helped |
| Add the BiMoT module (ablation) | DriveScape · nuScenes validation | FID: 20.98 -> 8.34 | Helped |
| Add the BiMoT module (ablation) | DriveScape · nuScenes validation | FVD: 142.81 -> 76.39 | Helped |
| Add temporal attention to BiMoT (ablation) | DriveScape · nuScenes validation | FID: 20.67 -> 8.34 | Helped |
| Add temporal attention to BiMoT (ablation) | DriveScape · nuScenes validation | FVD: 123.03 -> 76.39 | Helped |
| Add the key-frame condition (ablation) | DriveScape · nuScenes validation | FID: 11.60 -> 8.34 | Helped |
| Add the key-frame condition (ablation) | DriveScape · nuScenes validation | FVD: 108.80 -> 76.39 | Helped |
Surprising findings
- Drive-WM, using only an image model, achieved better FID than its full video model, illustrating the difficulty of video generation quality.
Limitations
- Relies on nuScenes annotations and cross-view conditioning design; key views must be non-adjacent for the scheme to work.
- Perception evaluation uses single-image inference, not exploiting temporal consistency of generated videos.
- Road mIoU lags DrivingDiffusion, which uses post-processing refinement.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: FID FVD road mIoU vehicle mIoU NDS
Baselines: BEVGen BEVControl MagicDrive DriveDreamer DrivingDiffusion Drive-WM Delphi SparseCtrl Panacea
Builds on: Stable Video Diffusion Latent Diffusion Models DrivingDiffusion MagicDrive SparseCtrl
Closest papers by meaning
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation (AAAI 2025 · similarity 0.89)
- Glad: A Streaming Scene Generator for Autonomous Driving (ICLR 2025 · similarity 0.88)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.88)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.88)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.87)
- SimGen: Simulator-conditioned Driving Scene Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.