Research map / Vision-language and multimodal / Multimodal LLMs

OpenView: Empowering MLLMs with Out-of-view VQA

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OpenView, a four-stage panorama-based pipeline that synthesizes multi-choice out-of-view VQA data, releases an OpenView-Dataset (~158k VQAs) for LoRA fine-tuning and a 1,327-item OpenView-Bench that scores choice and rationale accuracy.

Problem

Multimodal LLMs are evaluated mostly on content inside the image frame and struggle to reason about objects, activities, and scenes beyond the visible perspective view (out-of-view understanding).

Approach

Use full 360° panoramas as a grounded source to synthesize out-of-view multi-choice VQAs with a four-stage pipeline (panorama filtering, multi-view visual analysis, proposal generation with rationales, and confidence-based refinement with option shuffling and view jittering), then fine-tune MLLMs on it.

Key results

• Fine-tuning LLaVA-NeXT-Mistral-7B on OpenView-Dataset raises overall joint accuracy from 15.82% to 49.74%.
• Fine-tuned Qwen3-VL-8B-Instruct reaches 58.78% overall joint accuracy, up from 48.08%; Qwen2.5-VL-7B rises from 33.38% to 57.65%.
• Human participants achieve 83.0% choice accuracy versus the best model (GPT-5) at 62.25% overall choice accuracy and 61.79% joint accuracy.

Surprising findings

• Qwen3-VL-8B-Instruct rationale accuracy decreased after fine-tuning (90.88% to 84.05%) even though its joint accuracy improved.
• Many models achieve moderate choice accuracy but much lower rationale accuracy, suggesting guessing without coherent justification.

Limitations

• Human-level choice accuracy (83%) remains far above all models, so OOV understanding is far from solved.
• Synthetic data is generated entirely by a single assistant model (Qwen2.5-VL-72B), which may introduce hallucinated or biased content.
• Benchmark is relatively small (1,327 items) and relies on a GPT-based judge (DeepSeek V3.1) for rationale correctness.

Experimental setup

Datasets: Matterport3D Mapillary Metropolis 360Loc 360+x 360-1M OpenView-Dataset

Benchmarks: OpenView-Bench

Metrics: choice accuracy rationale accuracy joint accuracy

Baselines: GPT-4o GPT-4o mini GPT-5 Gemini-2.5-flash Qwen2.5-VL-72B-Instruct Qwen3-VL-8B-Instruct InternVL3.5-8B LLaVA-NeXT-Mistral-7B MiniCPM-V 4.5 Ovis2.5-9B

Builds on: Qwen2.5-VL LoRA MVDiffusion

Closest papers by meaning

• More than the Sum: Panorama-Language Models for Adverse Omni-Scenes (CVPR 2026 · similarity 0.90)
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.89)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.88)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.88)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
• ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments? (ICLR 2026 · similarity 0.88)