Research map / Vision-language and multimodal / Multimodal LLMs

PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PanoEnv-QA, a 14.8K-question panoramic (ERP) VQA benchmark built from TartanAir ground truth, benchmarks 14 VLMs, and post-trains Qwen2.5-VL-7B with LoRA via GRPO using ground-truth-routed rewards and a two-stage curriculum.

Problem

VLMs struggle with 3D spatial reasoning on equirectangular (ERP) 360° panoramas due to geometric distortion and a lack of 3D-grounded supervision and evaluation data.

Approach

Builds a panoramic VQA benchmark whose answers are programmatically derived from simulator 3D ground truth, then uses these answers as verifiable rewards in GRPO with question-type-routed reward functions and a structured-then-mixed two-stage curriculum.

Key results

• GRPO-Balanced (Qwen2.5-VL-7B + LoRA) reaches 52.93% total accuracy vs 49.34% for the base model, with OE accuracy rising from 6.39% to 14.83%.
• Among 14 zero-shot VLMs, the best total accuracy is 49.34% (Qwen2.5-VL-7B) and best OE accuracy is 8.36% (DeepSeek-VL2-Base and Qwen2.5-VL-32B).
• Zero-shot transfer to OSR-Bench: PanoEnv-RL (7B) scores 0.507 on object counting vs 0.498 for Qwen2.5-VL-72B, and 0.371 on relative distance vs 0.325.

Surprising findings

• Qwen2.5-VL-32B scores lower total accuracy than the 7B base model (42.70% vs 49.34%), so larger size did not help overall.
• The GRPO-Structured-only variant reaches 52.3% total accuracy but collapses to 5.7% on open-ended questions, showing that structured-only RL hurts generation.

Limitations

• Synthetic-to-real gap; evaluated on TartanAir-derived simulation data, with only a small zero-shot sim-to-real check on OSR-Bench.
• Open-ended accuracy remains low (14.83%) even after RL.
• Relative direction on OSR-Bench lags the 72B model, attributed to a 2D-vs-3D mismatch in that benchmark.

Experimental setup

Datasets: TartanAir PanoEnv-QA OSR-Bench

Benchmarks: PanoEnv-QA OSR-Bench

Metrics: accuracy open-ended accuracy Q-Score P-Score

Baselines: Qwen2.5-VL-7B Qwen2.5-VL-32B InternVL2.5-26B Qwen3-VL-8B DeepSeek-VL2-Base LLaVA-1.5-7B Llama3.2-11B-Vision

Builds on: GRPO DeepSeekMath PPO TartanAir OSR-Bench 360-R1

Closest papers by meaning

• PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World (NeurIPS 2026 · similarity 0.90)
• MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse (ICLR 2026 · similarity 0.89)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
• MindJourney: Test-Time Scaling with World Models for Spatial Reasoning (NeurIPS 2025 · similarity 0.88)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.88)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.88)