Research map / 3D vision / 3D generation
SceneCraft: Layout-Guided 3D Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates indoor 3D scenes (NeRF) from text and free-form bounding-box layouts by fine-tuning a layout-conditioned Stable Diffusion (SceneCraft2D) with semantic and depth ControlNets and distilling its multi-view outputs into Nerfacto.
Problem
Text- and layout-controlled generation of complex, multi-room indoor 3D scenes with free camera trajectories, which prior text-to-3D and panorama-based methods cannot support well.
Approach
Render user-specified bounding-box scenes into per-view semantic and depth maps (bounding-box images) that condition a fine-tuned Stable Diffusion via two ControlNets, then distill the generated views into a NeRF with annealed SDS-equivalent updates, a layout-aware depth loss, periodic migration to remove floaters, and a VGG perceptual loss.
Key results
- SceneCraft achieves CLIP Score 24.34 vs 23.85 (MVDiffusion) and 22.98 (Text2Room), the best among compared methods.
- User study (32 participants) gives SceneCraft 3.71 3D consistency and 3.56 visual quality, above Text2Room (3.11, 3.06) and MVDiffusion (3.20, 3.35).
- Set-the-scene and MVDiffusion have higher Inception Score (4.36 for MVDiffusion vs 3.54 for SceneCraft), which the authors attribute to fixed-category finetuning limiting diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SceneCraft instead of MVDiffusion | Indoor scene generation · Indoor scene generation | CS: 23.85 -> 24.34 | Helped |
| Use SceneCraft instead of MVDiffusion | Indoor scene generation · Indoor scene generation | IS: 4.36 -> 3.54 | Hurt |
| Use SceneCraft instead of Set-the-scene | Indoor scene generation · Indoor scene generation | 3DC: 3.53 -> 3.71 | Helped |
| Use SceneCraft instead of MVDiffusion | Indoor scene generation · Indoor scene generation | VQ: 3.35 -> 3.56 | Helped |
| Use a general base prompt instead of BLIP2 captions during finetuning (ablation) | SceneCraft2D · Layout-guided image generation | Layout-following ability | Helped |
| Add the layout-aware depth constraint during initial distillation (ablation) | SceneCraft · Indoor scene generation | 3D geometry correctness | Helped |
| Add texture consolidation with VGG perceptual and stylization loss (ablation) | SceneCraft · Indoor scene generation | Texture detail | Helped |
| Voxelize bounding boxes into finer-grained voxels for irregular layouts (ablation) | SceneCraft · Complex indoor scene layouts | Spatial representation | Helped |
Surprising findings
- Inception Score is lower for SceneCraft than MVDiffusion, which the authors attribute to fixed-category finetuning reducing diversity.
- Using BLIP2-generated captions as prompts during finetuning caused control failure, so a single generic base prompt was used instead.
Limitations
- Struggles with extremely complex layouts containing many closely placed or overlapping bounding boxes.
- Outputs are somewhat blurry for irregular objects such as hollowed chairs, lamps, and blinds.
- Mismatched layout and prompt (e.g., bedroom layout with kitchen prompt) leads to poor generation; only indoor scenes evaluated.
Experimental setup
Datasets: HyperSim ScanNet++
Benchmarks: Hypersim-layout scenes (qualitative) user study on generated rooms
Metrics: CLIP Score Inception Score 3D consistency (user study, 1-5) visual quality (user study, 1-5)
Baselines: Text2Room MVDiffusion Set-the-scene
Builds on: Stable Diffusion ControlNet Nerfacto Score Distillation Sampling IN2N
Closest papers by meaning
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.90)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.89)
- ControlRoom3D: Room Generation using Semantic Proxy Rooms (CVPR 2024 · similarity 0.89)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.89)
- Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model (AAAI 2025 · similarity 0.89)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.