Research map / 3D vision / 3D generation
SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SceneFactor generates large-scale text-guided 3D indoor scenes via a factored latent diffusion pipeline that first produces a coarse 3D semantic box layout and then geometry conditioned on it, enabling localized box-based editing.
Problem
Text-guided large-scale 3D scene generation that is also editable in localized regions without re-synthesizing the full scene.
Approach
Factor scene generation into a latent semantic diffusion model producing a proxy 3D semantic box map from text, followed by a latent geometric diffusion model conditioned on that map via convolutional spatial attention; edits are made by manipulating boxes and re-synthesizing only the affected latent region.
Key results
- On scene chunks, SceneFactor achieves CD MMD 0.021 and COV 0.410 versus SDFusion 0.031 and 0.282 (1-NNA CD 0.783 vs 0.882).
- In the neural listener text-consistency test, SceneFactor's output is preferred over SDFusion (58% vs 42%) and Text2Room (65% vs 35%).
- Ablations show removing the semantic stage, convolutional attention, or 3D latent space each degrades geometric quality metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proxy semantic generation stage rather than removing it (ablation) | SceneFactor · Scene chunks | MMD (CD): 0.026 -> 0.021 | Helped |
| Use convolutional attention rather than linear-layer attention (ablation) | SceneFactor geometric diffusion · Scene chunks | MMD (CD): 0.024 -> 0.021 | Helped |
| Use 3D latent grids rather than a 1D latent space (ablation) | SceneFactor · Scene chunks | MMD (CD): 0.025 -> 0.021 | Helped |
| Use SceneFactor instead of SDFusion | 3D scene generation · Independent chunks | MMD (CD): 0.034 -> 0.019 | Helped |
| Use SceneFactor instead of SDFusion | 3D scene generation · Scene chunks | MMD (CD): 0.031 -> 0.021 | Helped |
Surprising findings
- Applying the scene outpainting strategy to PVD and NFD failed to produce coherent scenes, so they were only compared on independent chunks.
Limitations
- Closed-vocabulary semantic layouts limit diversity of object types.
- Chunk-based outpainting makes room boundaries harder to control from text.
- Some baselines (PVD, NFD) fail under scene outpainting, so comparisons are limited.
Experimental setup
Datasets: 3D-FRONT 3D-FUTURE
Benchmarks: 3D-FRONT 3D-FUTURE
Metrics: MMD COV 1-NNA Chamfer Distance EMD neural listener accuracy perceptual user study
Baselines: PVD NFD SDFusion BlockFusion Text2Room
Builds on: DDPM VQ-VAE RePaint Latent Diffusion Models
Closest papers by meaning
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.90)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.89)
- Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation (CVPR 2025 · similarity 0.89)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.89)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.