Research map / 3D vision / 3D generation
DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DME-Driver, a two-part driving system where an LVLM (LLaVA-based) Decision-Maker produces human-like driving logic in language and a UniAD-based Executor converts it into planning outputs, trained with a new HBD dataset.
Problem
End-to-end planning-oriented driving systems lack interpretability of human driving logic, while LLM-centric driving systems do not exploit accurate 3D perception, so perception errors cannot be corrected through decision losses.
Approach
Decouple decision logic and perception: an LVLM Decision-Maker generates natural-language gaze, scene description, and decision reasoning, which conditions a UniAD-style Executor via text-encoded LogicalFusioner cross-attention on BEV features, plus an RL-style penalty for mismatch with the Decision-Maker's decision category.
Key results
- Planning: DME-Driver average L2 0.98 m and collision rate 0.29% versus UniAD 1.03 m and 0.31%.
- Decision-Maker logic understanding: DME-Driver scores 80.3 on logic versus GPT-4V 65.4 (ChatGPT-4 judged).
- Gaze score 85.2 for DME-Driver versus 75.3 for GPT-4V.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DME-Driver instead of GPT-4V | Driving logic understanding · HBD test | Gaze score: 75.3 -> 85.2 | Helped |
| use DME-Driver instead of GPT-4V | Driving logic understanding · HBD test | Scene Understanding score: 79.2 -> 86.5 | Helped |
| use DME-Driver instead of GPT-4V | Driving logic understanding · HBD test | Logic score: 65.4 -> 80.3 | Helped |
| integrate Decision-Maker guidance with the planning system | Planning · Planning, Avg. | L2 (m): 1.03 -> 0.98 | Helped |
| integrate Decision-Maker guidance with the planning system | Planning · Planning, Avg. | Collision rate (%): 0.31 -> 0.29 | Helped |
| add ground-truth language cues to the Executor (ablation) | Executor · Planning | L2 (m): 1.03 -> 0.94 | Helped |
| add ground-truth language cues to the Executor (ablation) | Executor · Planning | Collision rate (%): 0.31 -> 0.28 | Helped |
| add Decision-Maker guidance instead of ground-truth language cues (ablation) | Executor · Planning | L2 (m): 0.94 -> 0.96 | Hurt |
| add Decision-Maker guidance instead of ground-truth language cues (ablation) | Executor · Planning | Collision rate (%): 0.28 -> 0.28 | No clear effect |
| add Decision-Maker guidance to the Executor (ablation) | Executor · Planning | L2 (m): 1.03 -> 0.96 | Helped |
| add Decision-Maker guidance to the Executor (ablation) | Executor · Planning | Collision rate (%): 0.31 -> 0.28 | Helped |
| add consistency loss (ablation) | Decision-Maker + Executor · Planning | L2 (m): 0.96 -> 0.98 | Hurt |
Surprising findings
- Adding the consistency (RL-style) penalty slightly worsened planning metrics (L2 0.96 to 0.98, collision 0.28 to 0.29) while improving decision alignment.
Limitations
- Decision-Maker evaluation relies on ChatGPT-based scoring without human validation reported.
- Planning gains over UniAD are small (L2 1.03 to 0.98 m).
- Dataset annotations depend on GPT-4V pre-annotation with manual correction.
Experimental setup
Datasets: HBD (Human-Driver Behavior and Decision-Making) Look-Both-Ways BDD-X nuScenes Virtual HBD (CARLA) CC3M WebVid-10M LLaVA instruction data
Benchmarks: HBD test set nuScenes planning
Metrics: L2 (m) collision rate (%) ChatGPT-scored accuracy for gaze, scene understanding, logic
Baselines: LLaVA-7B LLaVA-13B GPT-4V NMP SA-NMP FF EO ST-P3 UniAD
Builds on: UniAD LLaVA BEVFormer RT-2
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.87)
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.86)
- SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.86)
- Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving (NeurIPS 2024 · similarity 0.86)
- $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.86)
- DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.