Research map / Video / Video generation
Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Genesis jointly generates multi-view driving videos and LiDAR sequences using a DiT video branch and a BEV-latent LiDAR diffusion branch, with DataCrafter VLM-based structured captions as shared conditioning, evaluated on nuScenes.
Problem
Driving scene generation typically handles only one modality (video or LiDAR), and existing multimodal approaches rely on costly occupancy labels and weak semantic supervision, leading to poor cross-modal consistency.
Approach
A two-branch joint generator in which camera and LiDAR diffusion share scene captions, layouts and conditions, with the LiDAR branch conditioned on LSS-lifted BEV features from generated images instead of occupancy grids, plus a VLM-based captioning module (DataCrafter) producing scene- and instance-level captions.
Key results
- Video generation with first-frame conditioning reaches FVD_multi 16.95 and FID_multi 4.24 on nuScenes, competitive with MiLA (18.20 / 3.00).
- LiDAR Chamfer@1s of 0.611 with ground-truth image conditioning vs. 0.78 for HERMES, and 0.633 at 3s vs. 1.17.
- Joint camera+LiDAR synthetic data raises BEVFusion mAP from 66.87 to 67.78 and NDS from 69.65 to 71.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adding DataCrafter (ablation) | Genesis video generation · nuScenes validation, w/o first cond | FVD_multi: 117.49 -> 85.91 | Helped |
| Adding DataCrafter (ablation) | Genesis video generation · nuScenes validation, w/o first cond | FID_multi: 22.32 -> 15.20 | Helped |
| Adding PoseNet to DataCrafter (ablation) | Genesis video generation · nuScenes validation, w/o first cond | FVD_multi: 85.91 -> 83.10 | Helped |
| Adding PoseNet to DataCrafter (ablation) | Genesis video generation · nuScenes validation, w/o first cond | FID_multi: 15.20 -> 14.90 | Helped |
| Adding the reference frame while retaining image BEV latent (ablation) | Genesis LiDAR generation · nuScenes validation, 1s prediction | Chamfer@1s: 0.668 -> 0.634 | Helped |
| Adding image BEV latent while omitting the reference frame (ablation) | Genesis LiDAR generation · nuScenes validation, 1s prediction | Chamfer@1s: 0.661 -> 0.668 | Hurt |
| Use Genesis instead of MagicDrive-V2 | Video generation · nuScenes validation, w/o first cond | FVD_multi: 94.84 -> 83.10 | Helped |
| Use Genesis instead of Drive-WM | Video generation · nuScenes validation, w/o first cond | FID_multi: 15.80 -> 14.90 | Helped |
| Use Genesis instead of MiLA | Video generation · nuScenes validation, w/ first cond | FID_multi: 3.00 -> 4.24 | Hurt |
| Use Genesis instead of UniScene | Video generation · nuScenes validation, w/ noisy latent | FVD_multi: 70.52 -> 67.87 | Helped |
| Use Genesis instead of UniScene | Video generation · nuScenes validation, w/ noisy latent | FID_multi: 6.12 -> 6.45 | Hurt |
| Use Genesis instead of HERMES | LiDAR generation · nuScenes validation, gt_img, 3s prediction | Chamfer distance: 1.17 -> 0.633 | Helped |
Surprising findings
- Video-only ablation without DataCrafter raises FVD_multi from 85.91 to 117.49 (reported in the ablation), showing caption sensitivity.
Limitations
- Training demands significant computational resources.
- Evaluated only on nuScenes; generalization to other domains requires adaptation.
- Real-time and interactive generation remains challenging.
Experimental setup
Datasets: nuScenes OmniDrive CoDALM
Benchmarks: nuScenes
Metrics: FVD FID Chamfer Distance mIoU mAP NDS
Baselines: MagicDrive Panacea DriveDreamer-2 MagicDrive-V2 Drive-WM MiLA UniScene HERMES ViDAR 4D-Occ DiVE Cogen
Builds on: MagicDrive CogVideoX rectified flow LSS BEVFusion
Closest papers by meaning
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.89)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.89)
- UniScene: Unified Occupancy-centric Driving Scene Generation (CVPR 2025 · similarity 0.89)
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.87)
- X-Drive: Cross-modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios (ICLR 2025 · similarity 0.87)
- DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.