Research map / 3D vision / 3D generation
Disentangled 3D Scene Generation with Layout Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates text-to-3D scenes decomposed into objects by jointly optimizing multiple NeRFs and multiple learnable layouts (per-NeRF affine transforms) under score distillation sampling from a pretrained image diffusion model (Imagen).
Problem
Text-to-3D generation produces monolithic representations that cannot be split into constituent objects; the task is to get unsupervised object disentanglement in generated 3D scenes using only a pretrained text-to-image prior.
Approach
Objects are defined as parts of a scene that can be rearranged in different ways and still yield valid compositions; the method optimizes K NeRFs plus N learnable layouts (per-NeRF affine transforms), sampling a random layout each step so composited scenes must look in-distribution to the image prior.
Key results
- Final layout learning achieves CLIP scores only 0.1 points below the per-object supervised upper bound when using the largest CLIP model as oracle.
- Ablation shows naive K NeRFs yield scattered objects with far-below-upper-bound CLIP scores, and multi-layout learning gives the largest gains.
- Method extends to conditional optimization with frozen 3D assets and to decomposing pre-existing NeRFs without per-object supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multiple NeRFs instead of a single NeRF (ablation) | Layout learning · 30 three-object prompts | CLIP score | Helped |
| Add per-NeRF regularization losses (ablation) | Multiple-NeRF scene generation · 30 three-object prompts | CLIP score | Helped |
| Add layout learning (ablation) | Multiple-NeRF scene generation · 30 three-object prompts | CLIP score | Helped |
| Learn multiple layouts instead of one layout (ablation) | Layout learning · 30 three-object prompts | CLIP score | Helped |
Surprising findings
- Viewpoint-dependent prompting was disabled because it does not aid compositional scene generation.
Limitations
- Inherits SDS failure modes such as bad geometry and Janus problem.
- Choosing the number of NeRFs K is difficult for prompts producing many small objects; objects that always move together may be grouped.
- Layouts sometimes converge to near-identical configurations.
Experimental setup
Datasets: 30 self-constructed prompts with three objects each
Benchmarks: 30-prompt CLIP NeRF-object matching evaluation
Metrics: CLIP score (100x cosine similarity)
Baselines: naive K NeRFs K NeRFs with regularization single-layout learning
Builds on: score distillation sampling Mip-NeRF 360 DreamFusion Instant NGP
Closest papers by meaning
- Towards Text-guided 3D Scene Composition (CVPR 2024 · similarity 0.89)
- GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs (CVPR 2024 · similarity 0.89)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.88)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.88)
- Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation (NeurIPS 2024 · similarity 0.88)
- SceneCraft: Layout-Guided 3D Scene Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.