Research map / Video / Video generation
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DriveDreamer-2, an LLM-driven pipeline that turns text prompts into agent trajectories, HDMaps, and multi-view driving videos via a unified multi-view video model (UniMVM) built on SVD.
Problem
Existing driving world models depend on structured conditions (3D boxes, HDMaps, optical flow) or real frames, limiting interactivity and the diversity of generated driving videos, especially for customized long-tail scenarios.
Approach
Fine-tune an LLM (GPT-3.5) on text-to-Python-script pairs built from a 18-function trajectory library to produce agent trajectories from text; generate background HDMaps conditioned on those trajectories with a diffusion model; and generate multi-view videos with UniMVM, which concatenates the six camera views into one spatially unified image and uses masking to support several generation tasks.
Key results
- Achieves FID 11.2 and FVD 55.7 on nuScenes validation with first-frame multi-view image condition, versus Panacea's 16.9 FID and 139.0 FVD.
- Generated data improves StreamPETR 3D detection mAP from 31.7 to 32.9 and NDS from 43.5 to 45.4 when used without initial-frame conditioning.
- Replacing SD1.4 with SVD and adding UniMVM reduces FVD from 340.8 (DriveDreamer) to 55.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DriveDreamer-2 instead of DriveDreamer without image conditioning | DriveDreamer video generation · nuScenes val, no image condition | FVD: 353.2 -> 105.1 | Helped |
| Use DriveDreamer-2 instead of Drive-WM with a 3-view video condition | multi-view video generation · nuScenes val, video condition | FVD: 122.7 -> 74.9; 39% relative improvement | Helped |
| Use DriveDreamer-2 instead of the strongest listed prior method | multi-view video generation · nuScenes val, first-frame multi-view image | FID: 14.9 -> 11.2 | Helped |
| Use DriveDreamer-2 instead of the strongest listed prior method | multi-view video generation · nuScenes val, first-frame multi-view image | FVD: 139.0 -> 55.7 | Helped |
| Add generated videos using the initial-frame condition to real training data | StreamPETR 3D detection · nuScenes | mAP: 31.7 -> 32.6; relative improvement of 2.8% | Helped |
| Add generated videos without image condition to real training data | StreamPETR 3D detection · nuScenes | mAP: 31.7 -> 32.9; relative improvement of 3.8% | Helped |
| Add generated videos using the initial-frame condition to real training data | StreamPETR multi-object tracking · nuScenes | AMOTA: 28.9 -> 31.2; relative enhancement of 8.0% | Helped |
| Add generated videos without image condition to real training data | StreamPETR multi-object tracking · nuScenes | AMOTA: 28.9 -> 31.3; relative enhancement of 8.3% | Helped |
| Replace SD1.4 with SVD while retaining the cross-view module (ablation) | DriveDreamer video generation · nuScenes val, first-frame multi-view image | FID: 14.9 -> 17.2 | Hurt |
| Replace SD1.4 with SVD while retaining the cross-view module (ablation) | DriveDreamer video generation · nuScenes val, first-frame multi-view image | FVD: 340.8 -> 94.6; almost 70% improvement | Helped |
| Add UniMVM instead of using the cross-view module (ablation) | DriveDreamer-2 with SVD · nuScenes val, first-frame multi-view image | FID: 17.2 -> 11.2 | Helped |
| Add UniMVM instead of using the cross-view module (ablation) | DriveDreamer-2 with SVD · nuScenes val, first-frame multi-view image | FVD: 94.6 -> 55.7 | Helped |
Surprising findings
- Without any image condition, DriveDreamer-2 gives better FVD (105.1) than DriveDreamer with its image condition (340.8) but worse FID than the Drive-WM variant, and the authors note the cross-view module slightly worsens FID under SVD.
Limitations
- Evaluation is limited to nuScenes and to FID/FVD plus one downstream perception model (StreamPETR).
- Trajectory generation relies on a hand-built 18-function library and manually curated script pairs, which constrains the scenario space.
Experimental setup
Datasets: nuScenes text-to-Python-script pairs (manually curated) trajectory-to-HDMap dataset (curated from nuScenes)
Benchmarks: nuScenes
Metrics: FID FVD mAP NDS AMOTA AMOTP IDS
Baselines: DriveDreamer Drive-WM DrivingDiffusion Panacea CTG++ LCTGen
Builds on: DriveDreamer Stable Video Diffusion ControlNet EDM
Closest papers by meaning
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.91)
- DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion (CVPR 2025 · similarity 0.89)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.89)
- Panacea: Panoramic and Controllable Video Generation for Autonomous Driving (CVPR 2024 · similarity 0.87)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.87)
- MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.