Research map / Vision-language and multimodal / Multimodal LLMs

GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GPT4Scene, a vision-only visual prompting paradigm that adds a BEV image and spatial-temporal object markers to video frames so VLMs can understand indoor 3D scenes, and fine-tunes Qwen2-VL-7B on a 165K-annotation ScanAlign dataset to reach SOTA on 3D QA, dense captioning, and visual grounding.

Problem

VLMs have limited 3D spatial comprehension of indoor scenes from video; existing strong 3D understanding relies on point clouds or multi-view image features aligned to LLMs, which is complex. The paper asks whether pure visual inputs can suffice.

Approach

Reconstruct the scene from video into a point cloud, render a bird's-eye-view (BEV) image for global layout, and overlay consistent Spatial-Temporal Object (STO) markers on both BEV and video frames so each frame's local objects correspond to the global map; use this visual prompt zero-shot for closed VLMs and as fine-tuning data (ScanAlign) for open VLMs.

Key results

• Fine-tuned Qwen2-VL-7B (GPT4Scene)-HDM reaches 60.6 EM-1 on SQA3D (vs 40.7 base and 54.6 for Chat-Scene) and 96.3 CIDEr on ScanQA.
• On Multi3DRef F1@0.5, Qwen2-VL-7B (GPT4Scene)-HDM scores 59.8 vs Chat-Scene's 52.4; on ScanRefer Acc@0.5 it reaches 57.0 vs Chat-Scene's 50.2.
• Zero-shot GPT-4o with GPT4Scene prompts improves ScanQA ROUGE from 32.6 to 37.7 and SQA3D EM-1 from 40.3 to 42.8, while small open models (2B, 7B, 8B) show limited or negative gains.

Surprising findings

• Zero-shot GPT4Scene gains are limited or negative for smaller open-source VLMs (InternVL2-8B, Qwen2-VL-2B), while larger models benefit more.
• Fine-tuned models keep improving at inference even without object markers or BEV image input, suggesting an intrinsic gain in 3D perception.

Limitations

• Zero-shot gains are limited for smaller VLMs, requiring fine-tuning on the custom ScanAlign dataset.
• Relies on 3D reconstruction and Mask3D instance segmentation to produce BEV images and markers, and evaluation uses Mask3D predicted boxes.
• Evaluated only on indoor ScanNet-based benchmarks; GPT Score evaluation uses a GPT-4o judge on 1,000 questions.

Experimental setup

Datasets: ScanNet ScanQA SQA3D Scan2Cap ScanRefer Multi3DRef MVBench MMBench MMStar RealWorldQA Video-MME

Benchmarks: ScanQA SQA3D Scan2Cap ScanRefer Multi3DRef MVBench MMBench MMStar RealWorldQA Video-MME

Metrics: BLEU-4 ROUGE CIDEr METEOR EM-1 EM-R1 IoU@0.25 IoU@0.5 Acc@0.25 Acc@0.5

Baselines: Chat-Scene Robin3D LEO LL3DA 3D-VisTA M3DRef-CLIP Scene-LLM Qwen2-VL-7B InternVL2-8B MiniCPM-V-2.6-8B

Builds on: Chat-Scene Mask3D Qwen2-VL BundleFusion

Closest papers by meaning

• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.92)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.91)
• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.91)