Research map / Vision-language and multimodal / Human and embodied modeling
Generative Planning with 3D-Vision Language Pre-training for End-to-End Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GPVL, which aligns BEV perception features with language via group-wise contrastive 3D-vision language pre-training and generates driving decisions and trajectories auto-regressively with a language model, evaluated on nuScenes.
Problem
Vision-based end-to-end autonomous driving struggles with vision understanding, decision reasoning and scene generalization, and existing LLM-injected methods use 2D features and slow LLM planning.
Approach
Aligns BEV-derived detection, motion and map features with BERT-encoded template descriptions via group-wise contrastive alignment, then feeds the aligned visual features, a captioning description and navigation prompt into a language model that generates the trajectory as text tokens.
Key results
- GPVL achieves average L2 of 0.43 m and collision rate of 0.14% on nuScenes open-loop planning, versus VAD at 0.72 m and 0.22%.
- Ablation shows removing the 3D-vision language pre-training and group-wise alignment raises average L2 from 0.43 to 0.79 and 0.62 m-level degradations in other configurations.
- Zero-shot cross-city tests (Boston to Singapore and reverse) and noisy-image tests show GPVL outperforms UniAD and VAD in L2 and collision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPVL instead of VAD | Open-loop planning on nuScenes val · nuScenes val | L2 Avg. (m): 0.72 -> 0.43 | Helped |
| Use GPVL instead of VAD | Open-loop planning on nuScenes val · nuScenes val | Collision Avg. (%): 0.22 -> 0.14 | Helped |
| Use GPVL instead of VAD | Planning with turn-left, turn-right, and go-straight commands · nuScenes val, turn left, turn right, and go straight | L2 Avg.: 0.85 -> 0.56; 0.90 -> 0.54; 0.70 -> 0.41 | Helped |
| Add perception modules (ablation) | GPVL · nuScenes val | L2 Avg.: 1.38 -> 0.43 | Helped |
| Add perception modules (ablation) | GPVL · nuScenes val | Collision Avg. (%): 0.75 -> 0.14 | Helped |
| Add captioning model (ablation) | GPVL · nuScenes val | L2 Avg.: 0.48 -> 0.43 | Helped |
| Add captioning model (ablation) | GPVL · nuScenes val | Collision Avg. (%): 0.19 -> 0.14 | Helped |
| Add 3D-vision language pre-training and group-wise alignment (ablation) | GPVL · nuScenes val | L2 Avg.: 0.79 -> 0.43 | Helped |
| Add 3D-vision language pre-training and group-wise alignment (ablation) | GPVL · nuScenes val | Collision Avg. (%): 0.38 -> 0.14 | Helped |
| Add group-wise alignment (ablation) | GPVL with 3D-vision language pre-training · nuScenes val | L2 Avg.: 0.62 -> 0.43 | Helped |
| Add group-wise alignment (ablation) | GPVL with 3D-vision language pre-training · nuScenes val | Collision Avg. (%): 0.29 -> 0.14 | Helped |
| Add cross-modal language model (ablation) | GPVL · nuScenes val | L2 Avg.: 0.56 -> 0.43 | Helped |
Surprising findings
- UniAD and VAD are argued to overfit to the dominant go-straight scenarios, underperforming on turning commands.
Limitations
- Open-loop evaluation only on nuScenes; no closed-loop testing reported.
- Relies on ground-truth template descriptions for textual alignment and pre-trained BLIP captioner trained on OmniDrive-nuScenes.
Experimental setup
Datasets: nuScenes OmniDrive-nuScenes
Benchmarks: nuScenes
Metrics: L2 displacement error collision rate latency FPS
Baselines: UniAD VAD ST-P3 Ego-MLP BEV-Planner LLaVA Vicuna Merlin Ominidrive Atlas
Builds on: BEVFormer VAD BLIP BERT
Closest papers by meaning
- VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · similarity 0.89)
- SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.88)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.87)
- $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.87)
- Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.87)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.