Research map / Vision-language and multimodal / Multimodal LLMs

OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 34 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniDrive, a counterfactual-reasoning-based synthetic 3D driving QA dataset generated from nuScenes with GPT-4, plus two VLM agents (Omni-L, Omni-Q) evaluated on DriveLM and nuScenes open-loop planning.

Problem

Vision-language models for autonomous driving lack full 3D understanding, and existing driving QA datasets provide sparse supervision mostly from expert trajectories, with open-loop planning evaluation having known limitations.

Approach

Generate 3D driving QA data by simulating alternative trajectories, checking them against rule-based checklists, and prompting GPT-4 with structured scene info to produce counterfactual reasoning and planning conversations; compare two VLM designs, Omni-L (LLaVA-style MLP projector) and Omni-Q (Q-Former with 3D position encoding and perception supervision).

Key results

• Omni-L with OmniDrive and LLaVA665k pre-training raises DriveLM score from 0.53 to 0.58 (accuracy 0.60 to 0.78).
• Training Omni-L without ego status with OmniDrive Q&A reduces nuScenes open-loop collision rate from 3.22% to 1.90% and intersection rate from 3.90% to 3.29%.
• Omni-L achieves counterfactual AP 53.7% and AR 63.0% with CIDEr 73.2, outperforming Omni-Q (52.3% AP, 59.6% AR, CIDEr 68.6).

Measured effects reported in the paper

Surprising findings

• Omni-Q's lower language ability and overfitting to ego status yields lower L2 error but higher collision rate than Omni-L.
• Training solely on trajectory prediction degrades open-loop planning for the LLM agent, while adding Q&A data mitigates this.

Limitations

• Counterfactual outcomes are simulated without modeling reactions of other agents, so closed-loop effects are not captured.
• Synthetic annotations rely on GPT-4 with human-in-the-loop checks only on sampled data.
• Omni-Q underperforms Omni-L due to weaker language ability, and BEV-MLP suffers from pre-training gaps.

Experimental setup

Datasets: nuScenes OpenLane-v2 DriveLM LLaVA665k

Benchmarks: DriveLM nuScenes open-loop planning OmniDrive counterfactual reasoning

Metrics: L2 error collision rate intersection rate precision recall CIDEr DriveLM score accuracy

Baselines: ST-P3 UniAD VAD-Base Ego-MLP BEV-Planner BEV-Planner++ DriveLM

Builds on: LLaVA BLIP-2 StreamPETR BEV-Planner DriveLM

Closest papers by meaning

• Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.90)
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.89)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.89)
• DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.89)