Research map / 3D vision / 3D generation

OneCanvas: 3D Scene Understanding via Panoramic Reprojection

NeurIPS 2026 · 3D vision › 3D generation · approach: Semantic scene completion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OneCanvas, which reprojects per-patch VLM features from posed RGB-D views onto one equirectangular panoramic canvas with a 3D position embedding, plus a synthetic spatial pretraining curriculum, reaching SOTA on SQA3D, VSI-Bench and SPBench with lower training compute.

Problem

3D scene understanding in VLMs typically requires dedicated geometry encoders or large spatial QA datasets, and models often ignore geometric input and rely on text/scene priors.

Approach

Lift each patch to 3D world coordinates using depth and pose, place it at its continuous longitude/latitude on a single equirectangular canvas centered on any chosen pose, add a metric 3D position embedding to the patch feature, and use procedurally generated synthetic object placements on an empty canvas as a spatial pretraining curriculum.

Key results

• OneCanvas reaches 65.3 EM@1 on SQA3D (2.3 points above prior best Ross3D at 63.0) with ground-truth geometry.
• OneCanvas reaches 71.3 average on VSI-Bench, above SpaceMind's 69.6.
• OneCanvas reaches 72.1 overall zero-shot on SPBench, 4.8 points above SpaceMind's 67.3.

Surprising findings

• Text-only baselines match or approach state-of-the-art geometry-aware models on widely cited situated 3D QA benchmarks, per audits cited by the paper.
• The base VLM with multi-view input retains a small lead on object counting over the full OneCanvas model (72.5 vs 68.9).

Limitations

• Requires depth and camera poses; pose-estimation failures can degrade the canvas.
• A single global panorama trades fine spatial precision for compactness and may limit very large or outdoor scenes.
• The pretraining curriculum is hand-authored, so new spatial skills require new task generators.

Experimental setup

Datasets: ScanNet ScanNet++ ARKitScenes EmbodiedScan SQA3D VSI-Bench VLM-3R-VSIBench ViCA SPBench

Benchmarks: SQA3D VSI-Bench SPBench

Metrics: EM@1 EM@R1 accuracy average accuracy mean absolute error

Baselines: SpaceMind SenseNova-SI Cambrian-S VLM-3R Ross3D SR-3D Loc3R-VLM GPT4Scene-HDM ViCA VST SpaceR-7B VILASR-7B

Builds on: Qwen3-VL 3D-RoPE LoRA Depth Anything 3

Closest papers by meaning

• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
• Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.90)
• Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.90)
• Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.89)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.89)
• PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World (NeurIPS 2026 · similarity 0.89)