Research map / 3D vision / 3D generation
InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructScene, a two-stage discrete semantic graph diffusion prior plus a continuous layout diffusion decoder that generates 3D indoor scenes from natural-language instructions, and curates an instruction-scene pair dataset from 3D-FRONT.
Problem
Instruction-driven 3D indoor scene synthesis is hard to control because existing methods model object joint distributions and encode relations implicitly, and they often ignore object appearance.
Approach
A semantic graph prior, learned by discrete masked (absorbing-state) diffusion over object categories, quantized OpenShape appearance features, and pairwise spatial relations, is conditioned on text; a separate graph-transformer diffusion decoder then produces continuous 7-DoF layouts from the graph, enabling zero-shot stylization, re-arrangement, completion and unconditional generation.
Key results
- On bedrooms, iRecall reaches 73.64% vs 56.43% for DiffuScene and 48.13% for ATISS, with FID 114.78 vs 119.73 (ATISS).
- On living rooms, iRecall reaches 56.81% vs 31.15% (DiffuScene) and 29.50% (ATISS), with lower FID (110.39 vs 117.67 for ATISS).
- Zero-shot re-arrangement on bedrooms reaches 79.59% iRecall vs 68.57% for DiffuScene; the paper reports ~15-25% iRecall gains across room types.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the semantic graph prior and layout decoder instead of DiffuScene | Instruction-driven scene synthesis · Bedroom | iRecall (%): 56.43\std{2.07} -> 73.64\std{1.37} | Helped |
| Use the semantic graph prior and layout decoder instead of DiffuScene | Instruction-driven scene synthesis · Living room | iRecall (%): 31.15\std{2.49} -> 56.81\std{2.85} | Helped |
| Use the semantic graph prior and layout decoder instead of DiffuScene | Instruction-driven scene synthesis · Dining room | iRecall (%): 37.87\std{2.76} -> 61.23\std{1.67} | Helped |
| Use the semantic graph prior and layout decoder instead of ATISS | Instruction-driven scene synthesis · Bedroom | FID: 119.73\std{1.55} -> 114.78\std{1.19} | Helped |
| Use the semantic graph prior and layout decoder instead of ATISS | Instruction-driven scene synthesis · Living room | FID: 117.67\std{2.32} -> 110.39\std{0.78} | Helped |
| Use the semantic graph prior and layout decoder instead of ATISS | Instruction-driven scene synthesis · Dining room | FID: 137.10\std{0.34} -> 129.76\std{1.61} | Helped |
| Use Gaussian diffusion for categorical graph attributes (ablation) | InstructScene · Bedroom | iRecall (%): 73.64\std{1.37} -> 34.18\std{2.53} | Hurt |
| Mask feature and relation attributes jointly with object categories (ablation) | InstructScene · Bedroom | iRecall (%): 73.64\std{1.37} -> 34.21\std{2.79} | Hurt |
| Use uniform transitions without mask states (ablation) | InstructScene · Bedroom | iRecall (%): 73.64\std{1.37} -> 69.22\std{3.25} | Hurt |
| Enforce permutation-invariant graph modeling (ablation) | InstructScene · Bedroom | iRecall (%): 73.64\std{1.37} -> 70.49\std{2.50} | Hurt |
| Reduce reverse diffusion timesteps (ablation) | InstructScene · Three room types | generation performance | No clear effect |
| Apply classifier-free guidance at an appropriate scale (ablation) | Diffusion-based scene synthesis · Three room types | iRecall (%) | Helped |
Surprising findings
- Permutation-invariant graph modeling was found to hurt performance (iRecall 70.49 vs 73.64), so the authors deliberately keep node order with random permutation augmentation.
- Gaussian and joint-mask transition matrices performed much worse than independent absorbing-state masking (iRecall about 34% vs 73.64%).
Limitations
- Only 3D-FRONT scenes (hundreds of scenes) are used, and 3D-FRONT contains misarrangements and misclassifications.
- Struggles with instructions containing more than 3 triplets, repeated object references, and abstract concepts such as style or function, attributed to the CLIP text encoder.
- Objects are retrieved from a database rather than generated, so the pipeline is not fully generative.
Experimental setup
Datasets: 3D-FRONT Objaverse (object retrieval)
Benchmarks: 3D-FRONT bedroom 3D-FRONT living room 3D-FRONT dining room
Metrics: iRecall FID FID-CLIP KID SCA Delta CLIP similarity (stylization)
Baselines: ATISS DiffuScene
Builds on: DiffuScene ATISS D3PM absorbing-state diffusion DDPM
Closest papers by meaning
- Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model (AAAI 2025 · similarity 0.92)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.91)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.91)
- FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts (CVPR 2025 · similarity 0.90)
- S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field (AAAI 2025 · similarity 0.89)
- Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.