Research map / 3D vision / 3D generation
Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DirectLayout, which generates numerical 3D indoor layouts from text via a three-stage LLM/VLM pipeline (BEV layout, 3D lifting, iterative refinement), trained with CoT activation (SFT) and DPO on a CoT-grounded dual-evaluator reward.
Problem
Text-to-3D indoor scene layout generation is limited by small datasets; existing methods either overfit to these datasets or rely on predefined symbolic constraints, which hurts flexibility, open-vocabulary coverage and alignment with fine-grained instructions.
Approach
Directly generate numerical layouts with LLM spatial reasoning, using a four-step chain-of-thought supervised via SFT, then DPO with a rule-based, entropy-weighted reward from a VLM (spatial) and a reasoning LLM (quantitative) evaluator, plus evaluator-driven iterative asset-layout alignment at inference.
Key results
- Ours (full) achieves 8.74 out-of-bound rate and 6.31 collision rate, versus 36.33 and 8.32 for the best-performing baseline on those metrics (Holodeck 36.33 OOB, LayoutGPT 15.89 collision)
- Ours (full) reaches Pos. 72.04, Rot. 69.87, PSA 75.56 and CLIP score 27.43, above LayoutGPT (57.24, 51.31, 55.11, 21.65)
- In the 25-person user study, Ours scores 4.56-4.84 on semantic compliance across granularities, versus at most 4.12 for baselines
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DirectLayout instead of I-Design | 3D indoor scene layout generation · 15 indoor scene categories | Out of Bound: 16.00 -> 8.74 | Helped |
| use DirectLayout instead of Holodeck | 3D indoor scene layout generation · 15 indoor scene categories | Collision: 8.32 -> 6.31 | Helped |
| use DirectLayout instead of LayoutGPT | 3D indoor scene layout generation · 15 indoor scene categories | Pos.: 57.24 -> 72.04 | Helped |
| use DirectLayout instead of LayoutGPT | 3D indoor scene layout generation · 15 indoor scene categories | Rot.: 51.31 -> 69.87 | Helped |
| use DirectLayout instead of LayoutGPT | 3D indoor scene layout generation · 15 indoor scene categories | PSA: 55.11 -> 75.56 | Helped |
| use DirectLayout instead of Holodeck | 3D indoor scene layout generation · 15 indoor scene categories | CLIP Score: 25.21 -> 27.43 | Helped |
| add staged 2D-to-3D task decomposition (ablation) | DirectLayout · 15 indoor scene categories | Out of Bound: 21.42 -> 8.74 | Helped |
| add staged 2D-to-3D task decomposition (ablation) | DirectLayout · 15 indoor scene categories | Collision: 20.52 -> 6.31 | Helped |
| add staged 2D-to-3D task decomposition (ablation) | DirectLayout · 15 indoor scene categories | PSA: 71.56 -> 75.56 | Helped |
| add the reinforcement reward signal (ablation) | DirectLayout · 15 indoor scene categories | Out of Bound: 26.04 -> 8.74 | Helped |
| add the reinforcement reward signal (ablation) | DirectLayout · 15 indoor scene categories | Collision: 16.77 -> 6.31 | Helped |
| add the reinforcement reward signal (ablation) | DirectLayout · 15 indoor scene categories | PSA: 68.82 -> 75.56 | Helped |
Surprising findings
- Replacing the specialized BEV generator with general reasoning model QwQ-32B lowers positional, rotational and semantic scores (Pos. 44.33 vs 72.04), though it keeps relatively low boundary violations
Limitations
- Task decomposition and iterative alignment add inference time (about 185 s average), limiting real-time interaction
- Scene complexity and number of small objects are bounded by base model capacity and training dataset scale and diversity
- BEV representation reduces clarity when objects are stacked vertically
Experimental setup
Datasets: 3D-Front GPT-4o-generated CoT annotations GPT-4o-generated scene descriptions
Benchmarks: 45 manually verified test prompts across 15 indoor scene categories (coarse/medium/fine)
Metrics: Out-of-Bound Rate Collision Rate Positional Coherency Rotational Coherency PSA CLIP Score human ratings for semantic compliance, physical plausibility
Baselines: LayoutGPT Holodeck I-Design QwQ-32B BEV variant w/o decomp w/o reward simple reward
Builds on: Direct Preference Optimization LayoutVLM ATISS LayoutGPT Trellis
Closest papers by meaning
- OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization (NeurIPS 2025 · similarity 0.92)
- Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model (AAAI 2025 · similarity 0.91)
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.91)
- LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models (CVPR 2025 · similarity 0.90)
- DeBaRA: Denoising-Based 3D Room Arrangement Generation (NeurIPS 2024 · similarity 0.90)
- SceneCraft: Layout-Guided 3D Scene Generation (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.