Research map / Vision-language and multimodal / Multimodal LLMs

Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows VLMs encode a weak latent 3D topology overshadowed by color/shape, extracts a spatial subspace by cross-scene averaging projection, and fine-tunes with a Dirichlet-energy regularizer (LoRA, 500 steps) to improve VSI-Bench and MindCube.

Problem

VLMs are unclear whether they form a topology-preserving 3D internal representation of scenes, and existing probes and VQA accuracy do not reveal it; spatial reasoning fails in cluttered multi-object scenes.

Approach

Isolate a spatial subspace by projecting out identity-attribute directions estimated via cross-scene averaging of object activations, show its principal components match Laplacian eigenmaps of a Gaussian-kernel 3D graph, and add the Dirichlet energy of this subspace as a single regularization term during LoRA SFT.

Key results

• Dirichlet-energy LoRA on Qwen2.5-VL-7B raises VSI-Bench overall accuracy from 29.3 (naive LoRA) to 35.6 and MindCube from 41.3 to 48.3.
• On InternVL3-8B, VSI-Bench overall accuracy rises from 36.7 to 42.4 with the proposed regularizer, versus 36.5 for 2D regularization and 36.0 for text-domain cognitive map prediction.
• Color-swap flips distance-order answers 41.0% of the time on Qwen2.5-VL-7B, nearly matching the 48.6% position-swap flip rate.

Surprising findings

• Color-swap, which leaves 3D positions unchanged, flips distance-order predictions almost as often as physically scrambling positions (41.0% vs 48.6%).
• Excessive regularization (lambda=9) drops Qwen2.5-VL-7B VSI-Bench accuracy to 28.6, below the unregularized baseline.

Limitations

• Training and probing use uniformly distributed synthetic scenes; real-world density is more complex, and the theory assumes uniform density.
• Gains are small on route planning and appearance order, which the authors attribute to temporal reasoning outside the regularizer's scope.
• Depth-shortcut confound remains for z-coordinate probing, acknowledged by the authors.

Experimental setup

Datasets: SynSpat3D VSI-Bench MindCube

Benchmarks: VSI-Bench MindCube

Metrics: accuracy Dirichlet ratio RSA answer flip rate probe R^2

Baselines: Naive LoRA fine-tuning 2D regularization Text-domain cognitive map prediction

Builds on: Laplacian eigenmaps linear probing linear representation hypothesis Ky Fan theorem

Closest papers by meaning

• HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.90)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.89)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.89)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)