Research map / Vision-language and multimodal / Multimodal LLMs

More than the Sum: Panorama-Language Models for Adverse Omni-Scenes

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Panorama-Language Modeling with a plug-and-play panoramic sparse attention module on Qwen2.5-VL and a 653K-QA panoramic driving VQA benchmark (PanoVQA) covering normal, occluded, and accident scenes.

Problem

Existing VLMs are built for pinhole images and handle 360-degree scenes by stitching multiple narrow-FoV views, losing holistic wrap-around context; dense attention on high-resolution equirectangular panoramas is also costly, and no large panoramic VQA benchmark covers adverse driving scenarios.

Approach

Adds a parallel panoramic sparse attention (top-K token selection via a gated learned indexer with position-aware gating) to the ViT alongside sliding-window attention, so pretrained pinhole VLMs can process single equirectangular panoramas after fine-tuning, plus a new panoramic VQA dataset built from NuScenes, BlendPASS and DeepAccident.

Key results

• PanoLM-7B (fine-tuned Qwen2.5-VL-7B with PSA) reaches 45.48 average GPT-score vs 45.21 for fine-tuned Qwen2.5-VL-7B and 35.56 for zero-shot Qwen2.5-VL-32B-Ins. on PanoVQA.
• On Qwen2.5-VL-3B ablation, PSA with frozen LLM reaches 32.14 vs 28.55 for LoRA baseline; unfreezing LLM gives 41.49 vs 41.42 for full SFT 3B.
• Panoramic (1-pano) input outperforms 6-camera multi-view input after SFT, 41.42 vs 40.22 average on PanoVQA-mini.

Surprising findings

• Zero-shot panoramic input underperforms 6-camera multi-view input for Qwen2.5-VL-3B (26.25 vs 28.26), while fine-tuned panoramic input wins.
• Frontier proprietary models GPT-5 and Claude-Sonnet-4.5 score relatively low on PanoVQA-N (around 19-22) despite strong averages on the occlusion and accident subsets.

Limitations

• Stitching to ERP requires vertical cropping and boundary distortion, causing pixel-level information loss
• Multi-view inputs outperform panoramic input in zero-shot settings and at high-resolution detail
• Occluded subset (PanoVQA-O) is very small (~1.3K QA), limiting statistical reliability

Experimental setup

Datasets: NuScenes BlendPASS DeepAccident

Benchmarks: PanoVQA

Metrics: normalized GPT score (1-100)

Baselines: Qwen2.5-VL-7B-Ins. InternVL3-8B-Ins. InternVL2-8B LLaVA-OneVision-7B LLaVA-NEXT-7B Chameleon-7B GLM-4-9B DeepSeek-VL2-Small Qwen2.5-VL-32B-Ins. Gemini-2.5-Flash-Lite Grok-3-Mini Qwen2.5-VL-7B (panoramic SFT)

Builds on: Qwen2.5-VL LoRA OneBEV Simplified Sparse Attention

Closest papers by meaning

• OpenView: Empowering MLLMs with Out-of-view VQA (NeurIPS 2026 · similarity 0.90)
• PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World (NeurIPS 2026 · similarity 0.88)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.87)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.86)
• PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning (CVPR 2026 · similarity 0.86)
• Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.86)