Research map / Image generation / Text-to-image diffusion
Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Supplementary material for Build-A-Scene, a training-free, guidance-free diffusion method for interactive 3D layout control and object consistency, comparing against Layout-Guidance and LooseControl.
Problem
Layout control for text-to-image diffusion models is limited to 2D layouts and static, pre-specified layouts; the appendix addresses how to move objects consistently in 3D while preserving their appearance.
Approach
Translate objects in 3D by rendering the bounding box's Cartesian coordinate maps before and after translation, warping the object into its new projected bounding box, and blending inverted latents to keep the scene coherent, without training or guidance.
Key results
- On 3D layout control, the proposed method reaches mIOU 0.807 versus 0.446 for Layout-Guidance and 0.712 for LooseControl.
- Layout-Guidance achieves higher object accuracy (52.3%) than the proposed method (46.8%) and LooseControl (25.1%), but adheres much less to the layout.
- On object consistency, the proposed method scores CLIP_I2I 0.940, SSIM 0.476, PSNR 29.5, above LooseControl and Layout-Guidance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Layout-Guidance instead of Build-A-Scene | 3D layout control · 3D Layout Control | CLIP_T2I: 0.321 vs 0.330 | Helped |
| Use Layout-Guidance instead of Build-A-Scene | 3D layout control · 3D Layout Control | mIoU: 0.807 vs 0.446 | Hurt |
| Use Layout-Guidance instead of Build-A-Scene | 3D layout control · 3D Layout Control | object accuracy (%): 46.8 vs 52.3 | Helped |
| Use Build-A-Scene instead of LooseControl | 3D layout control · 3D Layout Control | mIoU: 0.712 vs 0.807 | Helped |
| Use Build-A-Scene instead of Layout-Guidance | object consistency · Object Consistency | CLIP_I2I: 0.838 vs 0.940 | Helped |
| Use Build-A-Scene instead of Layout-Guidance | object consistency · Object Consistency | SSIM: 0.189 vs 0.476 | Helped |
| Use Build-A-Scene instead of Layout-Guidance | object consistency · Object Consistency | PSNR: 28.35 vs 29.5 | Helped |
| Use Build-A-Scene instead of LooseControl | object consistency · Object Consistency | SSIM: 0.367 vs 0.476 | Helped |
Surprising findings
- Layout-Guidance has the best object accuracy and CLIP_T2I but the worst layout adherence, suggesting guidance overweights object cross-attention at the expense of the layout.
Limitations
- Guidance-free design yields lower object accuracy than Layout-Guidance.
- Layout-Guidance ignores layout changes and fails object consistency under translation.
Experimental setup
Metrics: CLIP_T2I object accuracy (OA) mIOU CLIP_I2I SSIM PSNR
Baselines: Layout-Guidance LooseControl
Builds on: Layout-Guidance LooseControl
Closest papers by meaning
- Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint (ICLR 2025 · similarity 0.91)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.90)
- HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation (NeurIPS 2024 · similarity 0.90)
- SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation (AAAI 2024 · similarity 0.90)
- CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step (NeurIPS 2025 · similarity 0.90)
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.