Research map / Vision-language and multimodal / Multimodal LLMs

PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PanoWorld, an MLLM that reads ERP 360° panoramas natively via a capability-aligned metadata pipeline (570K panoramas, ~3M instruction samples) and a Spherical Spatial Cross-Attention adapter, evaluated on PanoSpace-Bench, H*Bench, and R2R-CE.

Problem

Multimodal LLMs struggle with spatial reasoning over 360° panoramas, which are typically decomposed into perspective views that leave ERP spherical geometry implicit; there is no systematic capability formulation or unified benchmark for panoramic MLLM understanding.

Approach

Treat the ERP panorama itself as the native observer-centered spatial representation: build a verified metadata graph (geometry, semantics, depth) to generate capability-aligned instruction data, and inject spherical geometry into visual patch tokens via cross-attention from sinusoidal spherical position tokens with a gated residual.

Key results

• On PanoSpace-Bench, PanoWorld reaches 56.5 overall vs 30.8 for its Qwen3.5-9B base and 37.2 for the best listed open baseline (Mimo-v2.5).
• On H*Bench under zero-shot ERP input, PanoWorld scores 56.1 overall vs 38.4 for the strongest perspective-view baseline (HVS-3B).
• On R2R-CE Val-Unseen with panorama-only input, PanoWorld-VLN reaches 54.3 SR and 52.1 SPL, improving over GridMM's 49.0 SR and 41.0 SPL.

Surprising findings

• Naive H*Bench fine-tuning on ERP input degrades performance (17.8 overall) relative to the base model's prompted result, suggesting target-task supervision alone is insufficient without pano-native priors.

Limitations

• Benchmark PanoSpace-Bench and training data release are conditional on license compatibility; the paper states only that code and data 'will be' released.
• Training uses a single epoch with a very low learning rate, and results rely on self-generated labels from detectors and MLLMs without human validation reported.
• The VLN and H*Bench comparisons use differing training data fractions (80% of R2R/RxR) and setups, complicating fair comparison.

Experimental setup

Datasets: Realsee3D 360+X R2R-CE H*Bench

Benchmarks: PanoSpace-Bench H*Bench R2R-CE Val-Unseen

Metrics: multiple-choice accuracy BFOV mIoU success rate HOS HPS NE OSR SR SPL

Baselines: GPT-4o Gemini-2.5-Pro Qwen3.5-9B Qwen3-VL-32B InternVL3.5-14B HVS-3B GridMM StreamVLN DreamWalker

Builds on: Qwen3.5 Thinking in 360 WeDetect

Closest papers by meaning

• PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning (CVPR 2026 · similarity 0.90)
• SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.89)
• OneCanvas: 3D Scene Understanding via Panoramic Reprojection (NeurIPS 2026 · similarity 0.89)
• ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.88)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.88)