Research map / Vision-language and multimodal / Multimodal LLMs
LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LayoutVLM, which uses a VLM to generate numerical object poses plus differentiable spatial relations from visually marked renderings, refined by self-consistent decoding and differentiable optimization, for open-universe 3D indoor layout generation.
Problem
Open-universe 3D layout generation from unlabeled assets and free-form language instructions requires both physical plausibility (no collisions, in-bounds) and semantic alignment, which LLM pose prediction and constraint-solving methods struggle to achieve jointly in dense scenes.
Approach
A scene layout representation pairing VLM-predicted initial object poses with five differentiable spatial relations (distance, on_top_of, align_with, point_towards, against_wall); self-consistent decoding keeps only relations satisfied by the predicted poses, and PGD optimizes poses with a DIoU collision loss. Visual prompting with coordinate grids and asset-direction marks improves VLM spatial gr
Key results
- Averaged over 11 room types, LayoutVLM improves PSA by 40.8 points over the best baseline I-Design (58.8 PSA vs. 18.0 for I-Design).
- Removing the constraint-based optimization drops CF from 81.8 to 75.8 and IB from 94.9 to 14.1, showing optimization is key for in-boundary placement.
- Fine-tuning open-source LLaVA-NeXT-Interleave on the proposed representation raises PSA from 6.8 (FT on numerical poses) to 39.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LayoutVLM instead of I-Design | 11-room open-universe layout generation · 11 room types (average) | PSA: 18.0 -> 58.8 | Helped |
| Use LayoutVLM instead of LayoutGPT | 11-room open-universe layout generation · 11 room types (average) | IB: 24.2 -> 94.9 | Helped |
| Use LayoutVLM instead of I-Design | 11-room open-universe layout generation · 11 room types (average) | CF: 76.8 -> 81.8 | Mixed |
| Add visual image input (ablation) | LayoutVLM · 11 room types | PSA: 48.2 $\pm$ 1.5 -> 58.8 $\pm$ 3.4 | Helped |
| Add self-consistency decoding (ablation) | LayoutVLM · 11 room types | PSA: 46.4 $\pm$ 2.4 -> 58.8 $\pm$ 3.4 | Helped |
| Add spatial constraints and optimization (ablation) | LayoutVLM · 11 room types | PSA: 6.7 $\pm$ 1.6 -> 58.8 $\pm$ 3.4 | Helped |
| Add visual asset marks (ablation) | LayoutVLM · 11 room types | PSA: 52.2 $\pm$ 5.0 -> 58.8 $\pm$ 3.4 | Helped |
| Add visual coordinate marks (ablation) | LayoutVLM · 11 room types | PSA: 46.0 $\pm$ 2.7 -> 58.8 $\pm$ 3.4 | Helped |
| Add all visual marks (ablation) | LayoutVLM · 11 room types | PSA: 43.0 $\pm$ 2.2 -> 58.8 $\pm$ 3.4 | Helped |
| Add numerical pose initialization (ablation) | LayoutVLM · 11 room types | PSA: 41.0 $\pm$ 1.7 -> 58.8 $\pm$ 3.4 | Helped |
| Fine-tune on LayoutVLM scene representation rather than numerical poses | GPT-4o · Residential rooms | PSA: 11.9 $\pm$ 0.9 -> 48.1 $\pm$ 1.8 | Helped |
| Fine-tune on LayoutVLM scene representation rather than numerical poses | LLaVA · Residential rooms | PSA: 6.8 $\pm$ 2.2 -> 39.5 $\pm$ 5.7 | Helped |
Surprising findings
- Fine-tuning GPT-4o on numerical poses alone drops its IB score to 29.6 and PSA to 11.9, below zero-shot GPT-4o (PSA 43.2).
Limitations
- Occasional failures to produce valid layouts due to suboptimal VLM initializations.
- VLMs struggle with overall layout coherence in spatial planning despite handling pairwise relations.
- Evaluation relies on GPT-4o-based semantic scores validated on a small human study.
Experimental setup
Datasets: 3D-Front Objaverse
Benchmarks: LayoutVLM open-universe test cases (11 room types, 3 rooms each)
Metrics: Collision-Free Score (CF) In-Boundary Score (IB) Positional Coherency Rotational Coherency PSA Kendall's Tau
Baselines: LayoutGPT Holodeck I-Design
Builds on: Holodeck LayoutGPT I-Design self-consistency
Closest papers by meaning
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.90)
- Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.90)
- S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.90)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.90)
- G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.90)
- Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.