Research map / Video / Video generation
DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an autonomous-driving world model (DriveDreamer) to synthesize novel-trajectory videos from altered ego actions and trains 4D Gaussian Splatting on temporally aligned real and generated data (Cousin Data Training Strategy) to improve novel-view driving scene rendering.
Problem
Existing NeRF and 3DGS driving scene reconstructions render well only near the training distribution (forward driving), and degrade on complex maneuvers such as lane change, acceleration and deceleration in closed-loop simulation.
Approach
Treat a driving world model as a data generator: propose safe novel ego trajectories (NTGM), render structured conditions (3D boxes, HDMap) for them, generate videos, and jointly optimize 4DGS on temporally aligned original/generated pairs with a perceptual regularization loss, dropping depth supervision for novel views.
Key results
- FID on Waymo lane change improves by 32.1%, 46.4%, and 16.3% over PVG, S3Gaussian, and Deformable-GS (105.29 to 71.52 for PVG).
- Average NTA-IoU rises by 22.6%, 43.5%, and 15.6% across the three 4DGS baselines on Waymo.
- User study win rate of DriveDreamer4D over baselines averages 84.1% to 96.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DriveDreamer4D to PVG | PVG · Waymo novel trajectory views (average) | NTA-IoU: 0.349 -> 0.428 | Helped |
| Add DriveDreamer4D to S3Gaussian | S3Gaussian · Waymo novel trajectory views (average) | NTA-IoU: 0.331 -> 0.475 | Helped |
| Add DriveDreamer4D to Deformable-GS | Deformable-GS · Waymo novel trajectory views (average) | NTA-IoU: 0.321 -> 0.371 | Helped |
| Add DriveDreamer4D to PVG | PVG · Waymo lane change | FID: 105.29 -> 71.52 | Helped |
| Add DriveDreamer4D to PVG | PVG · nuScenes lane change | NTA-IoU: 0.204 -> 0.338 | Helped |
| Add DriveDreamer4D to PVG | PVG · PandaSet lane shift 3m | FID: 76.8 -> 74.6 | Helped |
| Add novel-trajectory training with λnovel=0.5 instead of λnovel=0 (ablation) | DriveDreamer4D with PVG · Waymo lane change | NTA-IoU: 0.349 -> 0.405 | Helped |
| Set λnovel to 1 instead of 0 (ablation) | DriveDreamer4D with PVG · Waymo lane change | NTA-IoU: 0.349 -> 0.420 | Helped |
| Set λnovel to 1.5 instead of 0 (ablation) | DriveDreamer4D with PVG · Waymo lane change | FID: 105.29 -> 82.10 | Helped |
| Remove regularization loss (ablation) | DriveDreamer4D with PVG · Waymo lane change | FID: 71.52 -> 79.54 | Hurt |
| Set λreg to 1e-2 instead of 1e-3 (ablation) | DriveDreamer4D with PVG · Waymo lane change | FID: 71.52 -> 119.39 | Hurt |
| Remove depth loss for novel-trajectory views (ablation) | DriveDreamer4D with PVG · Waymo lane change | FID: 82.63 -> 79.54 | Helped |
Surprising findings
- Adding LiDAR depth loss for novel trajectories hurts performance, because projected LiDAR depth is incomplete due to occlusion from the original view.
Limitations
- FID comparisons restricted to lane change scenarios because acceleration and deceleration views are already close to ground truth.
- Depth supervision cannot be used for novel trajectories due to LiDAR occlusion, limiting geometric constraints.
Experimental setup
Datasets: Waymo nuScenes PandaSet
Benchmarks: Waymo nuScenes PandaSet
Metrics: NTA-IoU NTL-IoU FID user study win rate
Baselines: PVG S3Gaussian Deformable-GS UniSim NeuRAD
Builds on: 3D Gaussian Splatting Deformable-GS PVG S3Gaussian DriveDreamer
Closest papers by meaning
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.92)
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation (AAAI 2025 · similarity 0.91)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.89)
- ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration (CVPR 2025 · similarity 0.89)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.88)
- DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.