Research map / Vision-language and multimodal / Multimodal LLMs
SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOLVE, coupling a VLM (LLaVA-1.5-7B) and an end-to-end planner through a shared Q-Former visual encoder, trajectory chain-of-thought over a trajectory bank, and asynchronous trajectory-level handoff, evaluated on nuScenes open-loop planning.
Problem
End-to-end autonomous driving lacks common-sense reasoning and interpretability, while VLMs are computationally too expensive for real-time planning and existing VLM+E2E combinations only cooperate at the trajectory level.
Approach
A Sequential Q-Former shares compressed visual queries between a VLM and an E2E planner (feature-level synergy); the VLM refines trajectories coarse-to-fine by selecting from a k-means trajectory bank (T-CoT); and VLM trajectories are stored in memory and used asynchronously as initialization for the E2E planner (temporal decoupling).
Key results
- SOLVE-VLM reaches 0.28 m average L2 and 0.20% average collision rate on nuScenes open-loop planning, the best among listed methods.
- SOLVE-E2E (Async) reaches 0.30 m average L2 versus 0.31 m for SOLVE-E2E and 0.33 m for OmniDrive.
- Trajectory CoT lowers VLM average L2 from 29.5 cm (w/o CoT) to 28.4 cm with 6 reference trajectories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use historical VLM trajectories to initialize the E2E planner | SOLVE-E2E · nuScenes open-loop planning | L2 Avg. (m): 0.31 -> 0.30; improvement of 0.01 | Helped |
| Use 256 SQ-Former queries instead of 384 (ablation) | SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 32.3 -> 32.6 | Hurt |
| Use 512 SQ-Former queries instead of 384 (ablation) | SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 32.3 -> 32.6 | Hurt |
| Add image-feature encoding before detection and lane encoding (ablation) | SQ-Former in SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 33.0 -> 32.3; 0.7 performance improvement | Helped |
| Encode detection queries before lane queries (ablation) | SQ-Former in SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 32.6 -> 32.3; 0.3 L2 improvement | Helped |
| Add T-CoT to direct trajectory prediction (ablation) | SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 29.5 -> 28.4; 1.1 performance improvement | Helped |
| Use 4 candidate trajectories in T-CoT (ablation) | SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 29.5 -> 28.6; 0.9 improvement over the baseline | Helped |
| Increase T-CoT candidate trajectories from 6 to 8 (ablation) | SOLVE-VLM · nuScenes SOLVE-VLM planning ablation | L2 Avg. (cm): 28.4 -> 28.4 | No clear effect |
| Add shared SQ-Former features to the E2E planner (ablation) | SOLVE-E2E · nuScenes E2E planning ablation | L2 Avg. (cm): 32.5 -> 31.0; 1.5 cm reduction | Helped |
| Add shared SQ-Former features to the VLM planner (ablation) | SOLVE-VLM · nuScenes VLM planning ablation | L2 Avg. (cm): 29.0 -> 28.4; 0.6 improvement | Helped |
Surprising findings
- The paper reports that the sequential Q-Former with 384 queries outperforms the 512-query variant, suggesting more visual queries do not help.
Limitations
- VLM latency means its trajectories are delayed, relying on temporal offset handling.
- Evaluated only on open-loop nuScenes planning, not closed-loop driving.
- Gains over strong baselines are small (e.g., 0.01 m for the async variant).
Experimental setup
Datasets: nuScenes OmniDrive-nuScenes
Benchmarks: nuScenes open-loop planning
Metrics: L2 error (m) collision rate (%)
Baselines: UniAD VAD-Base AD-MLP BEV-Planner BEV-Planner++ DriveVLM OmniDrive EMMA DriveVLM-Dual
Builds on: StreamPETR OmniDrive DriveVLM LLaVA-1.5 SparseDrive AD-MLP
Closest papers by meaning
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.91)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.91)
- From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving (NeurIPS 2026 · similarity 0.90)
- $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.90)
- VLP: Vision Language Planning for Autonomous Driving (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.