Research map / Vision-language and multimodal / Multimodal LLMs

OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OmniSpace, a plug-and-play training-time geometry adaptation for driving MLLMs (Plücker camera-pose injection, epipolar cross-view attention, and VGGT feature distillation with temporal aggregation) that improves planning, risk detection, captioning, and generalization without test-time 3D models.

Problem

MLLMs for autonomous driving treat scenes as 2D tasks and are bottlenecked by weak depth estimation and cross-view correspondence; existing geometry-aware fixes rely on external 3D models at inference, adding cost and cascading failures.

Approach

Internalize geometry during training only: encode visual tokens with zero-initialized Plücker ray embeddings, restrict cross-view attention to epipolar lines, and distill temporally aggregated VGGT features into the MLLM's visual tokens, discarding the teacher at inference.

Key results

• On Bench2Drive closed-loop, OmniSpace with Qwen3-VL-4B reaches Driving Score 81.40 and Success Rate 65.52%, the best MLLM result in the table.
• On nuScenes open-loop planning, OmniSpace (Qwen2.5-VL 7B) reaches average L2 0.28 and avg collision 0.22%, improving over its baseline (L2 0.37).
• On NuInstruct risk detection, OmniSpace reaches MAE 2.01, accuracy 64.81, MAP 40.36 and BLEU 90.66, with Avg* 48.45 versus 44.81 for the fine-tuned baseline.

Surprising findings

• Naive ray encoding gives only limited gains over no ray encoding, while Plücker rays give clearly better intersection rates.
• Vanilla unconstrained cross-view attention does not consistently improve all planning metrics, whereas epipolar-restricted attention does.

Limitations

• Relies on accurate camera calibration for Plücker rays and epipolar constraints.
• Uses a single fixed teacher (VGGT) that is not trained on driving data; temporal averaging ignores ego-motion.
• Training-time teacher feature extraction is costly (82 h for N=4 on one A100).

Experimental setup

Datasets: nuScenes Bench2Drive NuInstruct OmniDrive DriveLM Argoverse 2 Waymo ONCE TruckScenes

Benchmarks: nuScenes Bench2Drive NuInstruct OmniDrive DriveBench

Metrics: L2 collision rate intersection rate driving score success rate MAE accuracy MAP BLEU CIDEr

Baselines: UniAD VAD-Base BEV-Planner ST-P3 EMMA DriveVLM ORION OmniDrive VGGDrive SpaceDrive SimLingo HERMES

Builds on: VGGT Spatial-MLLM Neural Light Fields LoRA VGGDrive

Closest papers by meaning

• SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera (NeurIPS 2026 · similarity 0.89)
• SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.89)
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning (CVPR 2025 · similarity 0.87)
• On the Generalization Capacities of MLLMs for Spatial Intelligence (ICLR 2026 · similarity 0.87)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.87)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.87)