Research map / Label-efficient and robust learning / Recognition and long-tail
DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DriveWorld, a world-model-based 4D pre-training framework that learns spatio-temporal BEV representations from multi-camera driving videos via a Memory State-Space Model, then fine-tunes on nuScenes with BEVFormer/UniAD.
Problem
Vision-centric autonomous driving pre-training relies on 2D or 3D pre-text tasks and ignores the temporal (4D) nature of driving scenes, so spatio-temporal representations remain under-explored.
Approach
A world model with a Memory State-Space Model combining a Dynamic Memory Bank (temporal latent dynamics with motion-aware layer norm) and Static Scene Propagation (spatial latent statics from a random past BEV frame), plus a text-based Task Prompt, pre-trained to predict current and future 3D occupancy and actions.
Key results
- DriveWorld pre-trained on OpenScene improves 3D detection mAP from 0.377 (BEVFormer with 2D pre-training) to 0.452 (+7.5%).
- With OpenScene pre-training, minADE drops by 0.10 m (0.71 to 0.61) for motion forecasting and AMOTA rises 5.3% in tracking.
- Planning average L2 error is reduced by 0.34 m (1.03 to 0.69) with OpenScene pre-training, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| 4D world-model pre-training (DriveWorld) on OpenScene | BEVFormer detector · nuScenes val 3D detection | mAP: +7.5% (0.377 to 0.452) | Helped |
| 4D world-model pre-training on OpenScene | UniAD · nuScenes val motion forecasting | minADE: -0.10 m (0.71 to 0.61) | Helped |
| 4D world-model pre-training on OpenScene | UniAD · nuScenes val planning | average L2: -0.34 m (1.03 to 0.69) | Helped |
| Dynamic Memory Bank added to SSP (ablation) | UniAD pre-training (RSSM/MSSM) · nuScenes val ablation | mAP: 0.429 to 0.425 | Hurt |
| Dynamic Memory Bank added to SSP (ablation) | UniAD pre-training · nuScenes val ablation | AMOTA: 0.365 to 0.370 | Helped |
| Motion-aware layer normalization added (ablation) | UniAD pre-training · nuScenes val ablation | mAP: 0.425 to 0.432 | Helped |
| Task Prompt added (ablation) | UniAD pre-training · nuScenes val ablation | mAP: 0.432 to 0.436 | Helped |
| Replace RSSM with MSSM framework (RSSM 1D latent) (ablation) | UniAD pre-training · nuScenes val ablation | mAP: 0.381 vs 0.416 without pre-training | Hurt |
| Static Scene Propagation added to MSSM (ablation) | UniAD pre-training · nuScenes val ablation | mAP: 0.381 (RSSM) to 0.429 | Helped |
| Pre-training data halved (50%) (ablation) | UniAD pre-training · nuScenes val ablation | mAP: 0.436 to 0.425 | Hurt |
| Fine-tuning data reduced to 75% (ablation) | DriveWorld fine-tuning · nuScenes val ablation | mAP: 0.436 to 0.418 | No clear effect |
| Pre-training on OpenScene instead of nuScenes only | UniAD · nuScenes val occupancy prediction | IoU-far: +5.0% (40.2 to 45.2) | Helped |
Surprising findings
- Adding the Dynamic Memory Bank alone lowered detection performance while improving tracking, the authors attribute this to a wide perceptual field adding noise to precise localization.
Limitations
- Annotations still depend on LiDAR point clouds rather than purely vision-based self-supervision.
- Validated only with a lightweight ResNet101 backbone and limited dataset scale.
Experimental setup
Datasets: nuScenes OpenScene
Benchmarks: nuScenes OpenScene
Metrics: mAP NDS AMOTA AMOTP IoU VPQ minADE minFDE MR L2
Baselines: BEVFormer with 2D ImageNet pre-training FCOS3D OccNet UniScene BEVDistill UniAD ST-P3 BEVerse
Builds on: Recurrent State-Space Model MILE UniAD BEVFormer StreamPETR
Closest papers by meaning
- UniPAD: A Universal Pre-training Paradigm for Autonomous Driving (CVPR 2024 · similarity 0.83)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.83)
- DualAD: Disentangling the Dynamic and Static World for End-to-End Driving (CVPR 2024 · similarity 0.83)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.83)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.83)
- SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.