Research map / 3D vision / 3D generation
Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Layout-Your-3D generates compositional 3D scenes from text by using a 2D layout to create a reference image, reconstructing per-instance 3DGS with LGM, then refining layout (collision-aware) and instances (timestep-adjusted SDS) in about 12 minutes.
Problem
Text-to-3D methods struggle to produce plausible object interactions and precise user control for compositional scenes, and optimization-based methods are slow.
Approach
Use a 2D layout-guided reference image (MIGC) to drive feed-forward per-instance 3DGS reconstruction (LGM), initialize 3D layout from 2D boxes plus DINOv2-based rotation estimation, then apply a collision-aware layout optimization followed by instance-wise SDS refinement with adjustable timestep sampling.
Key results
- On Compo20, Layout-Your-3D reaches 53.51% BLIP-VQA and 50.44% mGPT-CoT, the best among compared methods, with about 12 minutes per prompt versus 2 to 4 hours for GraphDreamer and ProlificDreamer.
- CLIP-Score is 23.26% for Layout-Your-3D versus 24.93% for ProlificDreamer, so ProlificDreamer leads on that metric.
- User study (500 responses from 100 users) gives Layout-Your-3D 9.08 text-alignment, 8.44 quality, and 8.72 rationality, above all baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Layout-Your-3D instead of GaussianDreamer | Compo20 text-to-3D generation · Compo20 | Time cost: 15 mins -> 12 mins | Helped |
| Use Layout-Your-3D instead of ProlificDreamer | Compo20 text-to-3D generation · Compo20 | BLIP-VQA Score: 42.14% -> 53.51% | Helped |
| Use Layout-Your-3D instead of GraphDreamer | Compo20 text-to-3D generation · Compo20 | mGPT-CoT: 44.28% -> 50.44% | Helped |
| Use Layout-Your-3D instead of ProlificDreamer | Compo20 text-to-3D generation · Compo20 | CLIP-Score: 24.93% -> 23.26% | Hurt |
| Use Layout-Your-3D instead of ProlificDreamer | Compo20 user study · Compo20 user study | Text-Alignment: 7.34 -> 9.08 | Helped |
| Use Layout-Your-3D instead of LucidDreamer | Compo20 user study · Compo20 user study | Quality: 7.72 -> 8.44 | Helped |
| Use Layout-Your-3D instead of ProlificDreamer | Compo20 user study · Compo20 user study | Rationality: 6.91 -> 8.72 | Helped |
| Use DINOv2 rather than no rotation estimation (ablation) | Layout-Your-3D coarse rotation estimation · Compo20 rotation estimation | Average error: 18.9$^{\circ}$ -> 5.5$^{\circ}$ | Helped |
| Use DINOv2 rather than CLIP features for coarse rotation estimation (ablation) | Layout-Your-3D coarse rotation estimation · Compo20 rotation estimation | Average error: 10.8$^{\circ}$ -> 5.5$^{\circ}$ | Helped |
| Add tolerant collision loss to layout refinement (ablation) | Layout-Your-3D · Toy example with overlapping instances | Layout coherence | Helped |
| Add feature-level reference loss to layout refinement (ablation) | Layout-Your-3D · Toy example with overlapping instances | Layout coherence | Helped |
| Adjust timestep sampling during instance-wise refinement (ablation) | Layout-Your-3D · 3D instance refinement | Geometry and appearance quality | Helped |
Surprising findings
- CLIP-Score did not track semantic correctness here, so the authors added BLIP-VQA and mGPT-CoT, and ProlificDreamer scored higher on CLIP-Score than Layout-Your-3D.
Limitations
- Relies on carefully designed 2D layouts and degrades for scenes with more than about 10 instances.
- The SSDS-based layout loss does not harmonize interactions when many instances are involved.
- Extremely complex or counter-intuitive prompts mislead the diffusion guidance.
Experimental setup
Datasets: Compo20 (self-constructed validation prompts)
Benchmarks: Compo20
Metrics: CLIP-Score BLIP-VQA score mGPT-CoT time cost user study ratings (text-alignment, quality, rationality) rotation average error
Baselines: DreamFusion ProlificDreamer GaussianDreamer LucidDreamer GraphDreamer
Builds on: DreamFusion LGM MIGC ComboVerse threestudio
Closest papers by meaning
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.91)
- GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting (ICML 2024 · similarity 0.90)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.88)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.88)
- Towards Text-guided 3D Scene Composition (CVPR 2024 · similarity 0.88)
- DecompDreamer: A Composition-Aware Curriculum for Structured 3D Asset Generation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.