Research map / Large language models / Agents, code and math
Distilling Multi-modal Large Language Models for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiMA, which jointly trains a vision-based end-to-end planner with an MLLM (LLaVA-v1.5-7B LLM) and distills the MLLM's knowledge into the planner via surrogate tasks, so the LLM can be dropped at inference.
Problem
LLM-based end-to-end autonomous driving planners are computationally expensive at test time, while vision-based planners struggle in long-tail driving scenarios.
Approach
Use the vision-based planner's scene encoder as a trainable tokenizer for an MLLM (structured BEAM tokens via Q-formers), jointly train both with VQA, planning, feature distillation and surrogate tasks (masked BEV reconstruction, future BEV prediction, scene editing), then discard the MLLM at inference.
Key results
- DiMA achieves a 37% reduction in L2 trajectory error and 80% reduction in collision rate versus the vision-based planner, with 44% trajectory error reduction in long-tail scenarios.
- Under VAD evaluation, DiMA (VAD-Tiny) outperforms VAD-Base by 47% in L2 error while running about 4x faster (16.8 vs 4.5 FPS).
- Under standardized evaluation, DiMA (VAD-Base) reaches 0.57 m average L2 and 0.06% collision on the full validation split, ahead of PARA-Drive and TOKEN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train VAD-Base with DiMA distillation | VAD-Base · nuScenes validation, standardized | Traj L2 Aveall (m): 0.78 -> 0.47 | Helped |
| train VAD-Base with DiMA distillation | VAD-Base · nuScenes validation, standardized | Collision Aveall (%): 0.30 -> 0.06 | Helped |
| train VAD-Tiny with DiMA | VAD-Tiny · nuScenes validation, VAD evaluation | Traj L2 Aveall (m): 0.78 -> 0.38 | Helped |
| use all BEAM scene tokens instead of BEV and map tokens (ablation) | DiMA · nuScenes validation | Traj L2 Aveall (m): 0.56 -> 0.52 | Helped |
| add feature distillation (ablation) | DiMA with all scene tokens · nuScenes validation | Traj L2 Aveall (m): 0.52 -> 0.48 | Helped |
| add masked token reconstruction (ablation) | DiMA with distillation · nuScenes validation | Traj L2 Aveall (m): 0.48 -> 0.42 | Helped |
| add future BEV prediction (ablation) | DiMA with masked reconstruction · nuScenes validation | Traj L2 Aveall (m): 0.42 -> 0.39 | Helped |
| add scene editing (ablation) | DiMA with future BEV prediction · nuScenes validation | Traj L2 Aveall (m): 0.39 -> 0.38 | Helped |
| train an LLM with planning and VQA using only BEV features (ablation) | VAD-Tiny · nuScenes validation | Traj L2 Aveall (m): 0.60 -> 0.62 | Hurt |
| use DiMA-Dual inference | DiMA (VAD-Tiny) · 3-point turn (zero-shot) | Traj L2 Aveall (m): 1.17 -> 1.04 | Helped |
Surprising findings
- Naively training an LLM jointly with VAD-Tiny on planning and VQA with only BEV features gives inconsistent gains and drops in performance (ablation ID-2).
Limitations
- Using the MLLM at inference (DiMA-Dual) reduces speed substantially (3.5 FPS), so the best hybrid results cost efficiency.
- Some baseline comparisons rely on reported results because code and models for prior MLLM planners are unavailable.
Experimental setup
Datasets: nuScenes DriveLM
Benchmarks: nuScenes open-loop planning
Metrics: L2 trajectory error (m) collision rate (%) latency (ms) FPS
Baselines: UniAD VAD-Base VAD-Tiny PARA-Drive TOKEN AD-MLP OmniDrive DriveVLM ST-P3
Builds on: VAD UniAD LLaVA-v1.5 DriveLM Q-former
Closest papers by meaning
- VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · similarity 0.89)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.88)
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.88)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.88)
- LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.