Research map / Vision-language and multimodal / Multimodal LLMs
VLP: Vision Language Planning for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLP, which uses a frozen pretrained language model (CLIP text encoder) as a training-only teacher via contrastive alignment of BEV agent features (ALP) and ego-query features (SLP) to improve end-to-end driving planning, generalization, and long-tail performance on nuScenes.
Problem
Vision-only end-to-end autonomous driving systems lack reasoning, generalize poorly to new cities and long-tail scenarios, and need better planning safety.
Approach
Use a frozen pretrained language model to supervise BEV agent features and the ego-vehicle planning query through contrastive alignment with textual descriptions of ground truth; the LM is used only during training, so inference adds no parameters or computation.
Key results
- VLP-VAD reduces average L2 error by 35.9% and collision rate by 60.5% relative to VAD on nuScenes open-loop planning (validation).
- VLP-UniAD reduces average L2 by 28.1% and collision rate by 48.4% relative to UniAD.
- In new-city generalization, VLP-VAD improves Singapore-trained-on-Boston and vice versa; e.g., Singapore avg L2 drops from 0.78 to 0.63 and collisions from 0.39 to 0.20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SLP and ALP | UniAD · nuScenes validation planning | Avg. L2 (m): 1.03 -> 0.74 | Helped |
| Add SLP and ALP | VAD · nuScenes validation planning | Avg. collision rate (%): 0.38 -> 0.15 | Helped |
| Add VLP | VAD · Boston↔Singapore new-city planning | Avg. L2 (m): 0.86 -> 0.73 | Helped |
| Add VLP | VAD · Singapore↔Boston new-city planning | Avg. collision rate: 0.39 -> 0.20 | Helped |
| Add VLP | UniAD · Long-tail nuScenes MOT | AMOTA: 29.6 -> 30.7 | Helped |
| Use SLP without ALP (ablation) | UniAD · nuScenes motion forecasting | minADE (m): 0.71 -> 0.72 | Hurt |
| Use SLP without ALP (ablation) | VAD · nuScenes vectorized scene segmentation | mIoU: 39.8 -> 47.2 | Helped |
| Use ALP without SLP (ablation) | VAD · VAD planning validation | Avg. L2: 0.78 -> 0.52 | Helped |
| Omit label ground truth from prompt (ablation) | VLP-UniAD · nuScenes planning prompts | Avg. L2: 0.52 -> 0.64 | Hurt |
| Omit bbox ground truth from prompt (ablation) | VLP-UniAD · nuScenes planning prompts | Avg. L2: 0.52 -> 0.56 | Hurt |
| Omit trajectory ground truth from prompt (ablation) | VLP-UniAD · nuScenes planning prompts | Avg. collision rate: 0.17 -> 0.36 | Hurt |
| Omit command ground truth from prompt (ablation) | VLP-UniAD · nuScenes planning prompts | Avg. L2: 0.52 -> 0.59 | Hurt |
Surprising findings
- CLIP ViT-L/14 and CLIP-RN101 text encoders give different trade-offs; ViT-L/14 raises collisions for UniAD in ablation (0.87 with LM-only integration), suggesting LM choice matters.
- The full VLP (SLP+ALP) does not always beat SLP-only on every metric, e.g., VLP-VAD mIoU drops from 47.2 (SLP) to 45.7 (SLP+ALP).
Limitations
- Experiments confined to nuScenes and camera-only baselines.
- Only CLIP-family and GPT-2 language models tested; LLaMA integration deferred to future work.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: L2 error collision rate AMOTA AMOTP IDS IoU VPQ minADE minFDE MR
Baselines: UniAD VAD NMP SA-NMP FF EO ST-P3 ViP3D QD3DT MUTR3D PnPNet FIERY
Builds on: UniAD VAD CLIP
Closest papers by meaning
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.91)
- SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.90)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Distilling Multi-modal Large Language Models for Autonomous Driving (CVPR 2025 · similarity 0.89)
- Generative Planning with 3D-Vision Language Pre-training for End-to-End Autonomous Driving (AAAI 2025 · similarity 0.89)
- LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.