Research map / 3D vision / 3D generation
Towards Text-guided 3D Scene Composition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneWiz3D, a hybrid explicit (DMTet objects) plus implicit (NeRF environment) 3D representation for text-to-3D scene generation, with PSO-based layout search and RGBD panorama diffusion guidance.
Problem
Generating high-fidelity 3D scenes containing multiple diverse, scattered objects from text, where unified representations and multi-view supervision are inadequate and layouts are hard to determine.
Approach
Model objects of interest with explicit DMTet and the rest of the scene with NeRF, optimize object layouts with particle swarm optimization guided by CLIP similarity instead of gradients, and add LDM3D RGBD panoramic score distillation plus MiDaS depth regularization to improve geometry.
Key results
- SceneWiz3D achieves CLIP-AP 97.6 versus 95.5 for ProlificDreamer and 92.6 for DreamFusion.
- Geometry FID drops to 10.43, against 11.33 for LDM3D and 76.22 for ProlificDreamer.
- Replacing PSO with gradient descent for layout lowers CLIP-AP from 97.6 to 91.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SceneWiz3D instead of the baseline methods | text-to-3D scene generation · 10 indoor test prompts | CLIP-AP: 97.6 vs. 95.5 (ProlificDreamer), 92.6 (DreamFusion), 69.3 (LDM3D), 63.4 (Text2room) | Helped |
| use SceneWiz3D instead of the baseline methods | text-to-3D scene generation · 10 indoor test prompts | align: 0.14 vs. 0.24 (LDM3D), 0.25 (DreamFusion), 0.30 (ProlificDreamer), 0.32 (Text2room) | Helped |
| use SceneWiz3D instead of the baseline methods | text-to-3D scene generation · 10 indoor test prompts | FID: 10.43 vs. 11.33 (LDM3D), 26.25 (Text2room), 41.18 (DreamFusion), 76.22 (ProlificDreamer) | Helped |
| add perspective-view guidance to panoramic-only guidance (ablation) | SceneWiz3D · 10 indoor test prompts | CLIP-AP: 59.7 -> 97.6 | Helped |
| add panoramic RGBD guidance (ablation) | SceneWiz3D with VSD perspective guidance · 10 indoor test prompts | FID: 68.53 -> 28.39 | Helped |
| add depth regularization (ablation) | SceneWiz3D with VSD perspective and panoramic guidance · 10 indoor test prompts | FID: 18.92 -> 10.43 | Helped |
| add PSO instead of using gradient descent for scene configuration (ablation) | SceneWiz3D · 10 indoor test prompts | CLIP-AP: 91.8 -> 97.6 | Helped |
| use VSD instead of SDS for perspective-view guidance (ablation) | SceneWiz3D · 10 indoor test prompts | CLIP-AP: 90.5 (SDS) -> 93.0 (VSD) without panoramic guidance; 96.7 (SDS) -> 96.1 (VSD) with panoramic guidance | Mixed |
Surprising findings
- SDS-based layout optimization does not find the optimal configuration, since the global minimum of the SDS score map does not match the best layout.
- Gradient-based layout updates fall into local minima, which motivates the switch to gradient-free PSO.
Limitations
- Long optimization time typical of SDS methods
- Color saturation and foggy or distorted geometry in some outputs
- CLIP-based layout optimization is weak for fine-grained manipulation
Experimental setup
Datasets: NYU-Depth-v2
Benchmarks: 10 indoor scene test prompts (100 views each, 1000 images)
Metrics: CLIP-AP disparity alignment MSE FID
Baselines: DreamFusion ProlificDreamer Text2Room LDM3D
Builds on: DreamFusion ProlificDreamer DMTet NeRF LDM3D
Closest papers by meaning
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.89)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.89)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs (CVPR 2024 · similarity 0.89)
- GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting (ICML 2024 · similarity 0.88)
- ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.