Research map / 3D vision / 3D generation
Global-Local Tree Search in VLMs for 3D Indoor Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a global-local tree search that prompts GPT-4o with emoji-labeled top-down grids and a hierarchical scene proxy to generate plausible 3D indoor layouts, outperforming HoloDeck and AnyHome on CLIP score and a user study.
Problem
Generating plausible 3D indoor scene layouts with VLMs is hard because token-level left-to-right reasoning cannot revise earlier placement decisions that cascade into unrealistic layouts.
Approach
Treat layout generation as a tree search: a global DFS places objects sequentially with backtracking, each object's placement is found by a local tree search over side, row, and column choices on a top-down grid whose cells are labeled with distinct emojis, and a hierarchical scene representation (room, region, floor object, supported object) reduces tree depth.
Key results
- Average CLIP score of the full method is about 28.4 vs HoloDeck and AnyHome, with 29.93 and 30.18 in bedroom and living room being best-of-method results.
- User study mean reciprocal rank of 0.793 average for the proposed method vs 0.596 for HoloDeck and 0.443 for AnyHome.
- Ablation: mean reciprocal rank 0.750 for full method vs 0.686 for CoT and 0.396 for IO prompting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use global-local tree search instead of direct input-output generation (ablation) | GPT-4o 3D scene generation · Average across scene types | mean reciprocal rank: 0.396 ±0.18 -> 0.750 ±0.26 | Helped |
| Use global-local tree search instead of chain-of-thought without candidate exploration (ablation) | GPT-4o 3D scene generation · Average across scene types | mean reciprocal rank: 0.686 ±0.26 -> 0.750 ±0.26 | Helped |
| Use global-local tree search instead of chain-of-thought (ablation) | GPT-4o 3D scene generation · Average across scene types | CLIP score: +0.21 | Helped |
| Use global-local tree search instead of chain-of-thought (ablation) | GPT-4o 3D scene generation · Bathroom | CLIP score: −0.46 | Hurt |
| Use global-local tree search instead of chain-of-thought (ablation) | GPT-4o 3D scene generation · Bedroom | CLIP score: −0.12 | Hurt |
| Use global-local tree search instead of AnyHome | 3D scene generation · Average across scene types | mean reciprocal rank: +0.360 | Helped |
| Use global-local tree search instead of HoloDeck | 3D scene generation · Average across scene types | mean reciprocal rank: +0.197 | Helped |
| Use global-local tree search instead of HoloDeck | 3D scene generation · Kitchen | CLIP score: less 0.24 CLIP score | Hurt |
Surprising findings
- The full tree search gives only a marginal CLIP gain over CoT (+0.21 on average) and is slightly worse than CoT in bathroom and bedroom, which the authors attribute to small branching factors k.
Limitations
- Small branching factors (k=1 or 2) may not explore the search space fully, reducing gains over CoT.
- Evaluation limited to 120 prompts across four room types and a 15-annotator user study.
- Relies on proprietary GPT-4o API, with API cost constraining search breadth.
Experimental setup
Datasets: Objaverse-1.0
Benchmarks: 120 generated prompts (bathroom, bedroom, kitchen, living room)
Metrics: CLIP score mean reciprocal rank (user study)
Baselines: HoloDeck AnyHome IO prompting (LayoutGPT-style) Chain-of-Thought
Builds on: Tree-of-Thoughts Chain-of-Thought LayoutGPT HoloDeck
Closest papers by meaning
- Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model (AAAI 2025 · similarity 0.89)
- SpatialLM: Training Large Language Models for Structured Indoor Modeling (NeurIPS 2025 · similarity 0.88)
- LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.87)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.87)
- OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization (NeurIPS 2025 · similarity 0.87)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.