Research map / Vision-language and multimodal / Multimodal LLMs

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GASP, which trains VLM LLM layers with a temporary correspondence head using point-correspondence InfoNCE and depth-consistency losses on DL3DV videos, improving spatial benchmarks without 3D VQA data.

Problem

VLMs lack robust 3D spatial reasoning; prevailing fine-tuning on 3D VQA datasets seems to encourage memorizing dataset biases and generalizes poorly, while specialized 3D encoders are cumbersome.

Approach

Inject geometric priors into intermediate LLM layers via a lightweight correspondence head supervised by ground-truth point correspondences (contrastive) and a depth-consistency loss on soft matching distributions; the head is discarded at inference so the model runs as a standard VLM.

Key results

• On All-Angles Bench camera pose estimation, Qwen2.5-VL-7B rises from 34.1 to 52.8 with the full GASP model.
• On VSI-Bench object counting, LLaVA-NeXT-Video-7B improves from 23.5 to 52.5 with GASP.
• Peak layer-wise correspondence PCK rises from below 5% to over 70%, with over 85% retained at 24-frame gaps.

Surprising findings

• A simple textual prior of average object and room sizes boosts baseline VLMs on VSI-Bench absolute distance (0.14 to 0.61), suggesting benchmark shortcuts.
• The DL3DV VQA fine-tuning baseline degrades several metrics relative to plain SFT.

Limitations

• Relies on pseudo ground-truth depth.
• Modest drop on NextQA (76.6 to 74.7) for Qwen2.5-VL-7B indicates trade-off on action-centric tasks.
• Evaluated only on 7B-scale models.

Experimental setup

Datasets: DL3DV LLaVA-Video-178K VGGT training collection

Benchmarks: All-Angles Bench VSI-Bench BLINK CV-Bench Video-MME TempCompass NextQA

Metrics: accuracy PCK@2 Pearson correlation of confidence and correctness normalized temporal PCK

Baselines: VG-LLM VLM-3R AoTD GPT-4o InternVL2.5-78B SFT on LLaVA-Video 178K SFT on DL3DV VQA

Builds on: VGGT DiffTrack InfoNCE LLaVA-Video-178K

Closest papers by meaning

• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
• Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models (CVPR 2026 · similarity 0.92)
• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.92)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.91)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
• SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (CVPR 2024 · similarity 0.91)