Research map / Vision-language and multimodal / Multimodal LLMs

UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniUGG, an LLM-based unified framework for spatial 3D understanding (VQA) and 3D scene generation, using a geometric-semantic pretrained ViT encoder, a Spatial-VAE, and a latent diffusion decoder conditioned on a reference view and Plücker raymap.

Problem

Unified understanding and generation of 3D scenes is largely unexplored; existing LLM-based models lack 3D-aware visual representations and tokenization for irregular 3D data make autoregressive 3D generation hard.

Approach

Pretrain a ViT-L encoder with RADIOv2.5 semantic distillation plus MASt3R-style multi-view geometric losses and RGB reconstruction; compress its features with a Spatial-VAE fine-tuned end-to-end with the spatial decoder; then have the LLM condition a latent diffusion model that generates target-view representations from a reference view and raymap.

Key results

• On VSI-Bench, UniUGG-3B reaches 40.1 average, reported as 17.9% above the second-best model in the paper's comparison.
• On 3D generation over ARKitScenes, UniUGG attains FID 55.01 and KID 0.0425, versus 64.16 FID for the RADIOv2.5-based variant and 138.54 for CUT3R.
• On the SPAR benchmark, UniUGG-3B averages 47.2 versus 38.1 for GPT-4o.

Surprising findings

• The paper reports its pretrained encoder does not outperform its RADIOv2.5 teacher on ImageNet-1K zero-shot and segmentation tasks, though it is competitive.
• Training generation directly on the original (non-compressed) representation without the Spatial-VAE fails to produce valid results.

Limitations

• No language-driven controllable generation or freeform editing of generated content.
• Does not support interactive multi-round scene generation and editing.
• Shows a performance gap versus 3D-enhanced methods on 3D benchmarks such as SQA3D, ScanQA, and ScanRefer.

Experimental setup

Datasets: ARKitScenes ScanNet++ LAION-400M LCS-558K ShareGPT4V ALLaVA SPAR EMOVA VSI-Bench BLINK SPAR-Bench SEED-Bench

Benchmarks: VSI-Bench BLINK 3DSRBench SPAR SEED-Bench RealWorldQA Feat2GS NYU-v2 BONN ImageNet-1K ADE20K PascalVOC2012

Metrics: accuracy PSNR SSIM LPIPS FID KID Abs Rel mIoU k-NN accuracy

Baselines: LLaVA-v1.5-7B LLaVA-NeXT-7B InternVL2.5-8B Qwen2.5-VL-7B GPT-4o Janus-Pro CUT3R LVSM RADIOv2.5-L MASt3R encoder DINOv2 CLIP

Builds on: MASt3R RADIOv2.5 stable diffusion Qwen2.5

Closest papers by meaning

• Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment (CVPR 2026 · similarity 0.89)
• A Unified Framework for 3D Scene Understanding (NeurIPS 2024 · similarity 0.89)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.89)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.89)
• Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.88)
• Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.88)