Research map / 3D vision / 3D generation
Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates hierarchical LLM scene descriptions (scene → functional areas → objects), infers fine-grained relative object placements with a hierarchy-aware variational graph network, and solves layouts via divide-and-conquer optimization for open-vocabulary indoor scene synthesis.
Problem
Converting LLM-generated open-vocabulary indoor scene descriptions into reasonable, physically feasible 3D layouts without overlap or out-of-boundary placements, while staying aligned with LLM-specified spatial relations.
Approach
Use a three-level hierarchical scene description (root, functional areas, objects) with sparse relations inside each area as LLM output, a CLIP-conditioned variational graph network to infer fine-grained relative placements, and a two-stage divide-and-conquer optimization (per-area then global layout).
Key results
- Achieves 0.00 OOB and 0.00 overlap on bedrooms and living rooms, with KL divergence 0.09 (bedroom) and 0.13 (living room), best among ATISS, DiffuScene, LayoutGPT, HOLODECK.
- Semantic alignment: #Rel. 0.88 vs 0.77 for HOLODECK on bedrooms; 0.86 vs 0.69 on living rooms.
- Perceptual study with 30 participants: highest scores on scene effectiveness (4.73), physical feasibility (4.69), and layout rationality (4.55).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the hierarchy-aware network for fine-grained relative-placement inference (ablation) | Hierarchical scene synthesis pipeline · Bedrooms | KL Div.: 0.23 -> 0.09 | Helped |
| add the hierarchy-aware network for fine-grained relative-placement inference (ablation) | Hierarchical scene synthesis pipeline · Living Rooms | KL Div.: 0.23 -> 0.13 | Helped |
| add divide-and-conquer layout optimization (ablation) | Hierarchical scene synthesis pipeline · Bedrooms | OOB: 0.73 -> 0.00 | Helped |
| add divide-and-conquer layout optimization (ablation) | Hierarchical scene synthesis pipeline · Living Rooms | OOB: 0.93 -> 0.00 | Helped |
| use hierarchical scene synthesis instead of HOLODECK | LLM-assisted scene synthesis · Bedrooms | KL Div.: 0.27 -> 0.09 | Helped |
| use hierarchical scene synthesis instead of HOLODECK | LLM-assisted scene synthesis · Living Rooms | KL Div.: 0.34 -> 0.13 | Helped |
| use hierarchical scene synthesis instead of HOLODECK | LLM-assisted scene synthesis · Bedrooms | OOB: 0.00 -> 0.00 | No clear effect |
| use hierarchical scene synthesis instead of HOLODECK | LLM-assisted scene synthesis · Living Rooms | Overlap: 0.00 -> 0.00 | No clear effect |
Surprising findings
- Removing the hierarchy-aware network while keeping divide-and-conquer optimization leaves KL divergence unchanged at 0.23, while the network alone reduces it but reintroduces overlap.
Limitations
- Assumes rectangular floor plans; irregular floors not handled.
- LLM sometimes produces infeasible configurations with too many or large objects, requiring random removal of areas or objects.
- Objects are represented as oriented bounding boxes, so object shape details (e.g., L-shaped sofas) are not captured.
Experimental setup
Datasets: 3D-FRONT Objaverse
Benchmarks: 3D-FRONT
Metrics: OOB overlap KL divergence semantic alignment (#Rel., #Obj.) perceptual study scores
Baselines: ATISS DiffuScene LayoutGPT HOLODECK
Builds on: LayoutGPT HOLODECK CLIP ATISS
Closest papers by meaning
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.92)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.91)
- S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field (AAAI 2025 · similarity 0.90)
- OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Global-Local Tree Search in VLMs for 3D Indoor Scene Generation (CVPR 2025 · similarity 0.89)
- SceneCraft: Layout-Guided 3D Scene Generation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.