Research map / Vision-language and multimodal / Multimodal LLMs
Can LVLMs Obtain a Driver’s License? A Benchmark Towards Reliable AGI for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IDKB, a ~1M-entry vision-language driving knowledge dataset built from handbooks, theory tests, and CARLA simulations, benchmarks 15 LVLMs on it, and fine-tunes several open models to show gains.
Problem
LVLMs lack explicit, structured driving knowledge (traffic laws, signs, driving techniques, defensive driving) because existing vision-language driving datasets focus on scene understanding and decision-making rather than explicit driving knowledge.
Approach
A driver's-license-inspired dataset combining handbook content, multi-country theory test questions (augmented with GPT-4o), and CARLA-rendered traffic-sign road scenes, used to benchmark and fine-tune LVLMs on driving knowledge and to transfer to nuScenes planning.
Key results
- GPT-4o achieved the highest IDKB score of 0.64; the best open-source model XComposer2 reached 0.45.
- Fine-tuning MiniCPM-Llama3-V2.5 on IDKB raised its IDKB score from 0.41 to 0.62 (42% improvement).
- Adding IDKB data to nuScenes fine-tuning of Qwen-VL-chat reduced average L2 by 32% (1.13 to 0.77 m) and collision rate by 65% (0.61% to 0.37%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune XComposer2 on IDKB | XComposer2 · IDKB test set | IDKB Score: 0.45 -> 0.64 | Helped |
| Fine-tune DeepSeek-VL-7B on IDKB | DeepSeek-VL-7B · IDKB test set | IDKB Score: 0.38 -> 0.60 | Helped |
| Fine-tune MiniCPM-Llama3-V2.5 on IDKB | MiniCPM-Llama3-V2.5 · IDKB test set | IDKB Score: 0.41 -> 0.62 | Helped |
| Fine-tune Qwen-VL-chat on IDKB | Qwen-VL-chat · IDKB test set | IDKB Score: 0.37 -> 0.56 | Helped |
| Add IDKB data to nuScenes planning fine-tuning | Qwen-VL-chat fine-tuned on nuScenes planning data · nuScenes validation | Average L2 (m): 1.13 -> 0.77 | Helped |
| Add IDKB data to nuScenes planning fine-tuning | Qwen-VL-chat fine-tuned on nuScenes planning data · nuScenes validation | Average Collision (%): 0.61 -> 0.37 | Helped |
Surprising findings
- Open-source LVLMs performed substantially worse on driving test (theory) questions than on road sign data, with most scoring around 0.25 on test data versus 0.44 on road data.
- Several open-source models had near-zero accuracy on multi-answer questions and poor instruction-following (e.g., Yi-VL-6B only 11%).
Limitations
- Much of the test-question data is augmented by GPT-4o and image descriptions are GPT-4o generated, with only manual screening for quality.
- Road data is limited to traffic-sign scenes simulated in CARLA, which may not transfer to real-world driving.
- Downstream evaluation is limited to a single Qwen-VL-chat model on nuScenes planning.
Experimental setup
Datasets: IDKB nuScenes Bench2Drive
Benchmarks: IDKB
Metrics: accuracy ROUGE-1 ROUGE-L SEMScore instruction-following rate L2 distance collision rate
Baselines: GPT-4o Gemini-1.5-flash VisualGLM-6B Qwen-VL-chat CogVLM BLIP2 LLaVA-v1.5-7B XComposer ShareGPT4V-7B mPLUG-Owl2 MiniCPM-Llama3-V2.5 XComposer2
Builds on: LoRA Agent-Driver DriveVLM CARLA UniAD
Closest papers by meaning
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.87)
- DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.86)
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.86)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.86)
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.85)
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.