Research map / Vision-language and multimodal / Multimodal LLMs

SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SGDrive, which structures an InternVL3-2B VLM's representations around scene, agent, and goal "world" queries with block-wise masked attention and a DiT diffusion planner, reaching state-of-the-art camera-only PDMS on NAVSIM.

Problem

VLMs used for end-to-end driving are generalist models that lack structured spatial-temporal understanding of 3D driving scenes, which hurts safe trajectory planning.

Approach

Introduces hierarchical world query tokens (scene geometry, safety-critical agents, short-term goal, plus future-state queries) decoded by auxiliary heads inside the VLM, with a block-wise structured attention mask that prevents leakage between knowledge categories, and conditions a DiT diffusion planner on these queries.

Key results

• SGDrive-2B reaches 87.4 PDMS on NAVSIM v1 navtest with SFT, above ReCogDrive-8B (86.8) and InternVL3-8B (83.3).
• With RFT, SGDrive-2B reaches 91.1 PDMS, versus 90.8 for ReCogDrive-2B.
• On NAVSIM v2, SGDrive-2B reaches 86.2 EPDMS, versus 83.6 for ReCogDrive-8B; it is slightly below DiffusionDrive on DAC, EP, LK, and EC.

Surprising findings

• Causal attention between world queries hurts planning by introducing cross-category noise, causing overly conservative driving.
• On NAVSIM v2, SGDrive-2B trails DiffusionDrive on DAC, EP, LK and EC, and trails ReCogDrive-8B on HC.

Limitations

• Uses only a single front-view camera, causing deviations under extreme turns and occasional lane-change errors.
• Stage-1 world knowledge learning and the planner are trained in separate stages with the VLM frozen, so the RFT extension is not tailored to the hierarchical world model.

Experimental setup

Datasets: NAVSIM navtrain NAVSIM navtest

Benchmarks: NAVSIM v1 navtest NAVSIM v2 navtest

Metrics: PDMS EPDMS NC DAC TTC EP Comf. TL LK EC

Baselines: ReCogDrive-8B ReCogDrive-2B AutoVLA-3B QwenVL2.5-8B InternVL3-8B DiffusionDrive WoTE SeerDrive Hydra-MDP TransFuser

Builds on: InternVL3 DiffusionDrive DETR ReCogDrive DiT

Closest papers by meaning

• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.92)
• DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.92)
• Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.91)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.90)
• $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.90)