Research map / Vision-language and multimodal / Multimodal LLMs

VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VGGDrive, which injects cross-view 3D features from a frozen VGGT into each decoder layer of Qwen2.5-VL-7B via a Cross-View 3D Geometric Enabler (cross-attention with camera-aware keys/values), improving VLM performance on NAVSIM, NuInstruct, DriveLM, OmniDrive, and nuScenes planning.

Problem

VLMs lack cross-view 3D geometric understanding, which limits their performance on autonomous driving perception, prediction, and planning tasks; existing ways of adding 3D foundation model features to VLMs are simple and tailored to indoor scenes.

Approach

A plug-and-play Cross-View 3D Geometric Enabler (CVGE) with per-layer, independently parameterized multi-head cross-attention lets 2D visual tokens at each LLM decoder layer query frozen VGGT 3D features (augmented with camera embeddings from img2lidar transforms), with the 3D-enhanced tokens replacing the visual hidden states via residual injection.

Key results

• On NAVSIM, VGGDrive reaches PDMS 88.76 vs. 86.04 for the Qwen2.5-VL-7B baseline (+2.72) and 86.68/86.10 for VGGT-Dist/VGGT-Add.
• On NuInstruct, MAP improves from 6.15 (baseline) to 37.49, surpassing the best prior method by 7.37 points.
• On DriveLM, Match rises from 34.54 to 49.77 and the average from 54.59 to 61.26, exceeding the best SOTA by 1.24 average points.

Surprising findings

• Injecting 3D features into a single decoder layer already yields PDMS around 88, close to full-layer injection, and peak performance appears near layer 11.
• Simply replacing the visual encoder with VGGT features (ID-2 ablation) degrades NAVSIM PDMS (84.45) and NuInstruct MAP (3.34) relative to the baseline.

Limitations

• Gains over VGGT-Dist/VGGT-Add on NAVSIM are modest in absolute terms (e.g., EP 82.92 vs 81.30), and comparisons rely on single-run results.
• Added inference latency (1.04 s vs 0.81 s) from per-layer cross-attention.
• Evaluated only with Qwen2.5-VL-7B and a single 3D expert (VGGT, with Fast3r as an alternative).

Experimental setup

Datasets: NAVSIM NuInstruct DriveLM OmniDrive nuScenes

Benchmarks: NAVSIM navtest NuInstruct DriveLM OmniDrive nuScenes open-loop planning

Metrics: PDMS EP NC DAC TTC Comfort MAE Accuracy MAP BLEU

Baselines: VGGT-Dist VGGT-Add DiffusionDrive WoTE ImagiDrive AutoVLA ReCogDrive AdaThinkDrive InMLLM FSDrive HERMES OmniDrive

Builds on: VGGT Qwen2.5-VL 3DRS

Closest papers by meaning

• Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.90)
• VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.90)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving (CVPR 2026 · similarity 0.88)
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving (NeurIPS 2025 · similarity 0.88)