Research map / Vision-language and multimodal / Multimodal LLMs
SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SafeAuto, which fine-tunes Video-LLaVA with a position-dependent cross-entropy loss for numeric control outputs, adds Markov Logic Network safety verification from traffic rules, and a multimodal RAG retriever, improving BDD-X and DriveLM results.
Problem
MLLM-based autonomous driving struggles with precise low-level control prediction when numbers are emitted as text, and with unsafe or rule-violating high-level actions due to hallucination and purely data-driven decisions.
Approach
A PDCE loss that replaces one-hot digit targets with Gaussian soft targets and place-weighted KL divergence so text-number prediction behaves more like MSE while keeping autoregressive generation; plus an MLN that encodes traffic rules as first-order logic to verify and override MLLM actions, and a joint video/control/predicate embedding trained to match text-embedding rankings for retrieval.
Key results
- On BDD-X high-level action, SafeAuto reaches CIDEr 337.4 versus 260.8 for RAGDriver (CIDEr 337.4 vs 260.8).
- On DriveLM high-level behavior accuracy, SafeAuto reaches 74.60 versus 61.60 for DriveLM-Agent.
- On DriveLM motion prediction, ADE drops to 0.84 from 1.51 for DriveLM-Agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Position-Dependent CE (PDCE) loss (ablation) | Video-LLaVA 7B low-level control prediction · BDD-X speed/course | RMSE: speed 0.76->0.63; course 4.18->3.89 | Helped |
| Add Multimodal RAG (top-K retrieval with predicate vectors) (ablation) | Video-LLaVA 7B high-level action · BDD-X action predicate accuracy | Accuracy: 61.94 -> 91.00 (with PDCE) | Helped |
| Add MLN post-safety verification (ablation) | Video-LLaVA 7B high-level action · BDD-X action predicate accuracy | Accuracy: 61.94 -> 62.97 (with PDCE) | Helped |
| Add MLN post-safety verification (ablation) | Video-LLaVA 7B high-level behavior · DriveLM behavior accuracy | Acc: 63.21 -> 66.86 (PDCE to PDCE+MLN) | Helped |
| Add Multimodal RAG (ablation) | Video-LLaVA 7B high-level behavior · DriveLM behavior accuracy | Acc: 63.21 -> 74.01 (PDCE to PDCE+RAG) | Helped |
| Add PDCE loss (ablation) | Video-LLaVA 7B on DriveLM · DriveLM motion ADE | ADE: 0.86 -> 0.85 | No clear effect |
| Add Multimodal RAG on top of PDCE (ablation) | Video-LLaVA 7B on BDD-X low-level control · BDD-X speed/course RMSE | RMSE: speed 0.63->0.65 (worse); course 3.89->3.85 (better) | Mixed |
| Add PDCE loss plus MLN and RAG (full SafeAuto) | Video-LLaVA 7B · BDD-X justification | CIDEr: 96.0 vs RAGDriver 109.1 | Hurt |
| Add full SafeAuto (PDCE+MLN+RAG) | Video-LLaVA 7B on BDD-X high-level action · BDD-X action CIDEr | CIDEr: 337.4 vs RAGDriver 260.8 | Helped |
| Add PDCE+MLN+RAG (full SafeAuto) | Video-LLaVA 7B on DriveLM · DriveLM high-level behavior accuracy | Acc: 74.60 vs DriveLM-Agent 61.60 | Helped |
| Add MLN and RAG to PDCE-trained model, reducing traffic-rule violations (ablation) | Video-LLaVA 7B on BDD-X · BDD-X rule violation rate | violation rate: 11.64% -> 4.50% | Helped |
| Remove environmental predicates from retrieval (keep only MLLM action predicates) (ablation) | Multimodal RAG on BDD-X action · BDD-X action predicate accuracy | Accuracy: 49.88 (all env. predicates only) vs 92.18 (with MLLM action predicates and all env. predicates) | Helped |
Surprising findings
- Justification performance on BDD-X is slightly lower than RAGDriver and ADAPT, with BLEU4 9.4 vs 11.4 for ADAPT.
- On DriveLM, PDCE and RAG do not reduce the traffic-rule violation rate of the base MLLM, which is attributed to mostly straight-line scenes.
Limitations
- Predicate extraction quality limits safety verification, especially with few predicates.
- Justification quality lags the strongest baselines on BDD-X.
- Low-level gains from MLN and RAG are minimal; most low-level improvement comes from PDCE.
Experimental setup
Datasets: BDD-X DriveLM LISA nuScenes California Driver Handbook
Benchmarks: BDD-X DriveLM
Metrics: BLEU4 CIDEr METEOR RMSE tolerant accuracy accuracy ADE
Baselines: ADAPT TimeLLM DriveGPT4 RAGDriver DriveLM-Agent UniAD BLIP-RT-2
Builds on: Video-LLaVA RAGDriver Markov Logic Network CLIP
Closest papers by meaning
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.90)
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.90)
- nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- World Knowledge-Enhanced Reasoning Using Instruction-Guided Interactor in Autonomous Driving (AAAI 2025 · similarity 0.88)
- DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution (NeurIPS 2026 · similarity 0.88)
- DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.