Research map / Video / Action recognition
DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DrivingDojo, a ~17.8k-clip driving video dataset from fleet data with complete ego maneuvers, multi-agent interplay, and open-world rare events, plus an action instruction following (AIF) benchmark and an SVD-based action-conditioned baseline.
Problem
Existing driving datasets used for world models are perception-oriented and lack diverse ego maneuvers, multi-agent interactions, and rare open-world events, limiting action-controllable driving world models.
Approach
Curate a large action-balanced, interaction-focused and long-tail driving video dataset from fleet data with camera poses, and define an AIF benchmark that estimates ego trajectories from generated videos via COLMAP and compares them against input action instructions.
Key results
- Fine-tuning SVD on DrivingDojo gives FID 19.20 and FVD 343.91 on OpenDV-2K, beating nuScenes (21.05/395.04) and no fine-tuning (24.17/580.94).
- In-domain AIF errors are 0.100m longitudinal and 0.062m lateral, versus 0.036m/0.019m for real GT images.
- Zero-shot on OpenDV-2K, the DrivingDojo-trained model achieves AIF errors of 0.238m (x) and 0.136m (y), better than ONCE (0.255m/0.23m) and nuScenes (0.387m/0.254m).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SVD on DrivingDojo instead of nuScenes | SVD · OpenDV-2K | FID: 21.05 -> 19.20 | Helped |
| Fine-tune SVD on DrivingDojo instead of nuScenes | SVD · OpenDV-2K | FVD: 395.04 -> 343.91 | Helped |
| Train on DrivingDojo instead of ONCE | SVD-XT world model · OpenDV-2K, zero-shot | FID: 28.37 -> 24.27 | Helped |
| Train on DrivingDojo instead of nuScenes | SVD-XT world model · OpenDV-2K, zero-shot | AIF lateral error: 0.254m -> 0.136m | Helped |
| Use out-of-domain rather than in-domain action sequences | DrivingDojo baseline model · DrivingDojo | AIF longitudinal error: 0.100m -> 0.173m | Hurt |
| Use out-of-domain rather than in-domain action sequences | DrivingDojo baseline model · DrivingDojo | AIF lateral error: 0.062m -> 0.110m | Hurt |
Surprising findings
- The model trained on ONCE always generates straight-line motion even when instructed to turn or change lanes, reflecting scarce lateral maneuvers in that dataset.
- Models show hallucinations such as objects suddenly disappearing and imagining new roads when actions are unrealistic.
Limitations
- Only single front-camera videos are included, reducing sensor diversity.
- The baseline is a plain SVD adaptation without specialized design, and only short (30-frame) videos are generated.
- Hallucinations occur, such as sudden object disappearance and new roads generated for unrealistic actions.
Experimental setup
Datasets: DrivingDojo nuScenes ONCE OpenDV-2K
Benchmarks: DrivingDojo AIF benchmark OpenDV-2K
Metrics: FID FVD AIF E_x AIF E_y
Baselines: SVD trained on nuScenes SVD trained on ONCE SVD trained on OpenDV-2K
Builds on: Stable Video Diffusion EDM COLMAP
Closest papers by meaning
- ReSim: Reliable World Simulation for Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024 · similarity 0.88)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.88)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.87)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.86)
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.