Research map / 3D vision / 3D generation

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

NeurIPS 2026 · 3D vision › 3D generation · approach: Open-vocabulary 3D scene understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Imagine3D-LLM appends learnable Gaussian summary tokens to a multi-view MLLM, decodes them into 3D Gaussians supervised by photometric loss plus distillation from a frozen compact Gaussian teacher (ZipSplat), and trains jointly with next-token prediction to improve 3D spatial reasoning.

Problem

Multi-view 3D reasoning remains weak in MLLMs; prior methods inject fine-grained pixel-level 3D signals or fuse geometry-foundation features, yielding only incremental gains.

Approach

Instead of pixel-aligned 3D signals, the MLLM learns a compact set of summary tokens after image tokens that are decoded into a coarse 3DGS scene representation via photometric reconstruction; the bottleneck (fewer summary tokens than image tokens) induces object-level grouping and propagates cross-view correspondence into the LLM's image features.

Key results

• Imagine3D-LLM reaches 63.8 EM on SQA3D test and 68.5 average on SPAR-Bench, versus 63.3 for 3DThinker-7B and 63.0 for Ross3D on SQA3D.
• On Scan2Cap CIDEr it reports 99.2 versus 86.1 for 3DRS and 81.3 for Ross3D.
• Controlled baseline with identical data and schedule improves SQA3D from 56.5 to 63.8 and SPAR from 60.9 to 68.5.

Surprising findings

• Feeding teacher query tokens directly to the LLM or distilling them without reconstruction loss gives no gain over baseline.
• Reconstruction training alone raises cross-view correspondence and semantic structure in the LLM's image features even though reconstruction loss is applied only to summary tokens.

Limitations

• Training efficiency: without the pretrained compact Gaussian teacher, more training steps are needed for comparable performance.
• Reconstruction quality is lower than the teacher (PSNR 17.61 vs 20.60 on SQA3D test), and the fixed Gaussian budget may not suit larger or outdoor scenes.
• Trained only on indoor scenes.

Experimental setup

Datasets: SQA3D ScanQA Scan2Cap ScanRefer Multi3DRefer SPAR-7M

Benchmarks: SQA3D Real-3DQA ScanQA Scan2Cap ScanRefer Multi3DRefer SPAR-Bench

Metrics: EM EM-R BLEU-4 METEOR ROUGE CIDEr Acc@0.25 Acc@0.5 F1@0.25 F1@0.5

Baselines: Video-3D-LLM Ross3D 3DRS LLaVA-3D ChatScene LEO VLM3R-7B 3DThinker-7B G2VLM-SR-2B Scene-LLM LL3DA GPT-4o

Builds on: LLaVA-Video ZipSplat C3G 3D Gaussian Splatting VLM3R

Closest papers by meaning

• Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views (CVPR 2026 · similarity 0.92)
• Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.92)
• MLLMs Need 3D-Aware Representation Supervision for Scene Understanding (NeurIPS 2025 · similarity 0.92)
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.91)
• Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.91)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)