Research map / Vision-language and multimodal / Multimodal LLMs

Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VLM3D, which uses a Qwen2.5-VL-7B Yes/No log-odds over multi-view renders as a differentiable semantic and geometric critic, applied as an SDS reward and as test-time guidance for Hunyuan3D sampling.

Problem

Text-to-3D generation suffers from coarse semantic alignment (missing prompt details) and poor 3D spatial understanding (geometric inconsistencies, failed part assembly), in both optimization-based and feed-forward pipelines.

Approach

A dual-query VLM prompt (content match and geometric quality over multi-view renders) whose Yes/No log-odds is backpropagated as a differentiable reward, used both as an added reward in SDS optimization and as test-time guidance during native 3D diffusion sampling.

Key results

• On GPTEval3D (110 prompts), VLM3D reports the highest Elo on all six metrics, e.g., Alignment 1365.5 and Overall 1268.6 versus DreamDPO's 1298.9 and 1203.1 (alignment is actually higher for DreamDPO; VLM3D leads the reported Alignment column at 1365.5 vs MVDream 1270.5).
• For feed-forward test-time guidance on 24 prompts, VLM3D lowers CLIP-D to 0.19 (vs Hunyuan3D 0.23) and FID to 274.9 (vs 338.6).
• In the VLM-only ablation, VLM3D reaches CLIP score 0.317 versus 0.290 for SDS-only MVDream.

Surprising findings

• VLM-only optimization without SDS already yields text-aligned 3D content (CLIP 0.271).

Limitations

• Struggles with very long or highly detailed prompts (misses finer details such as nurse's lifted leg in the Embracing Peace statue).
• High memory cost (49-53 GB) mainly from the VLM.
• Evaluation relies on GPT-4o-mini pairwise judging and a non-standard small feed-forward prompt set (24 prompts).

Experimental setup

Datasets: GPTEval3D prompts

Benchmarks: GPTEval3D CLIP-D/FID/CLIP-FID/Geometry on 24 feed-forward prompts ULIP Uni3D

Metrics: Elo rating (alignment, plausibility, T-G coherency, geo deta CLIP-D FID CLIP-FID Geometry score ULIP-T/I Uni3D-T/I CLIP score

Baselines: DreamFusion DreamGaussian Latent-NeRF ProlificDreamer MVDream DreamReward DreamDPO CLAY Hunyuan3D-2.1

Builds on: score distillation sampling MVDream Hunyuan3D 2.0 DreamReward

Closest papers by meaning

• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
• 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.90)
• G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.90)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.90)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.90)
• Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.90)