Research map / Vision-language and multimodal / Multimodal LLMs

Think Before You Drive: World Model-Inspired Multimodal Grounding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ThinkDeeper, a world-model-inspired visual grounding framework for autonomous driving that rolls out future latent states and fuses them with a hypergraph decoder, and releases the DrivePilot dataset built with RAG and CoT LLM annotation.

Problem

Visual grounding of natural-language driving commands is hard under ambiguous, context-dependent instructions, lacking 3D spatial reasoning and anticipation of scene evolution, while VLM-based approaches are too costly for real-time use.

Approach

A Spatial-Aware World Model distills the scene and command into a command-aware latent state, rolls out future latent states conditioned on depth-derived priors, and a cross-modal hypergraph decoder fuses these states with text for grounding.

Key results

• ThinkDeeper reaches 76.64 IoU0.5 on Talk2Car test, improving about 7.9% over the best baseline UNINEXT per the paper.
• On the Long-text DrivePilot subset, ThinkDeeper scores 74.08 IoU0.5, +5.28 over VLTVG.
• Ablation: removing the future-state rollout drops IoU0.5 by 11.7%, and removing the whole SA-WM drops it by 14.57 points to 62.70.

Surprising findings

• General-purpose VLMs such as Qwen2.5-VL-72B score well below specialized grounding models on driving grounding (e.g., 56.17 vs 76.64 on Talk2Car test).
• ThinkDeeper trained on only 50% of the data still outperforms most baselines trained on full data across test sets.

Limitations

• VLM baselines were evaluated without task-specific fine-tuning, so the comparison may not reflect their best achievable performance.
• DrivePilot annotations are generated by Qwen2-VL and only checked manually, which may propagate LLM errors.
• Future-state rollout is in latent space, so the predicted states are not directly interpretable as physical trajectories.

Experimental setup

Datasets: DrivePilot Talk2Car MoCAD RefCOCO RefCOCO+ RefCOCOg nuScenes

Benchmarks: DrivePilot Talk2Car MoCAD RefCOCO RefCOCO+ RefCOCOg

Metrics: IoU0.5 inference time

Baselines: AttnGrounder CMSVG TransVG CMRT MDERT VL-BERT RSD-LXMERT VLTVG UNINEXT CAVG MiniGPT-v2 LLaVA-NeXT

Builds on: TransVG VLTVG CenterNet ZoeDepth BERT

Closest papers by meaning

• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.89)
• DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.89)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.88)