Research map / Vision-language and multimodal / Multimodal LLMs

SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SphMind, a training-free plug-and-play framework that pairs a spherical-harmonics spatial graph (SHSG) with inference-time geometric grounding (IGG) to improve frozen VLMs' spatial reasoning on 360° panoramas.

Problem

Frozen MLLMs trained on perspective images struggle with directional and distance reasoning on equirectangular 360° panoramas due to distortion and wrap-around discontinuity.

Approach

Decouple semantic perception from geometry: encode object nodes with real spherical harmonics (rotation-equivariant), score directional candidates via Wigner-D rotation of the anchor, and steer the VLM's final hidden state with a norm-preserving Riemannian update minimizing a contrastive geometric energy, without weight updates.

Key results

• Across three backbones, SphMind raises overall accuracy on MP3D/Stanford2D-3D by about +20.5% (35.9% to 56.4%) with the largest direction gains for Qwen2.5-3B.
• On ODI-Bench, SphMind reaches 56.35% relative direction vs 46.13% baseline, beating prompt-engineering strategies by at least +8.7%.
• Full rotational consistency score rises from 0.043 (ERP-pixel) to 0.253 (SphMind), a x5.9 improvement.

Surprising findings

• Random geometric costs with gradients perform worse than no gradient at all, indicating uninformative costs actively misdirect the hidden state.
• Cartesian coordinates break equivariance and underperform spherical harmonics, and ERP bbox centroids perform worse still.

Limitations

• Relies on object detector (YOLO) and cubemap detections; errors propagate into SHSG.
• Evaluation on synthetic QA derived from 3D annotations for most results; real-world evaluation is limited.
• Some reported numbers are inconsistent across sections (e.g., Qwen2.5-3B direction baseline ranges and overall gains), making results hard to verify.

Experimental setup

Datasets: Matterport3D Stanford2D-3D-S ODI-Bench OpenView-VQA

Benchmarks: Matterport3D VQA Stanford2D-3D-S VQA ODI-Bench OpenView-VQA

Metrics: accuracy rotation consistency score (RCS)

Baselines: Qwen2.5-VL vanilla LLaVA-1.6 vanilla viewpoint guidance crop grounding response refinement chain-of-thought

Builds on: PPLM GrAInS Equiformer

Closest papers by meaning

• Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.90)
• SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.90)
• Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views (CVPR 2026 · similarity 0.89)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.89)
• GeoSense: Internalizing Geometric Necessity Perception for Multimodal Reasoning (ICML 2026 · similarity 0.89)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.89)