Research map / 3D vision / 3D generation
SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneWeaver, a reflective LLM/MLLM agent that iteratively selects from extensible 3D scene synthesis tools, reflects on GPT-4-scored feedback, and applies physics-aware Blender/Infinigen execution to generate indoor scenes.
Problem
Indoor 3D scene synthesis for embodied AI needs visual realism, physical plausibility, and controllability across open-vocabulary room types, but existing data-driven, rule-based, and LLM-based methods each fall short and pipelines are static with fixed tool sets.
Approach
A reason-act-reflect agent (built on OpenManus, ReAct-style) with a standardized tool-card interface for heterogeneous scene generators, a self-reflective planner that uses MLLM-scored feedback to choose the next tool and roll back when quality drops, and a physics-aware executor (Infinigen/Blender) that resolves collisions and boundary violations.
Key results
- On open-vocabulary generation, SceneWeaver averages 36.5 objects with 0 collisions and 0 out-of-boundary objects, versus 22.3 for Holodeck and 14.3 for I-Design.
- On common room types, SceneWeaver reaches 14.0 (bedroom) and 17.3 (living room) objects with zero collisions and out-of-boundary objects, and the highest realism, functionality, layout, and completeness scores except Holodeck's object count.
- In human evaluation, SceneWeaver is preferred in roughly 85% of pairwise comparisons; in simulation shift tests it achieves 1.0% objects moved >0.1m versus 17.6% for Holodeck.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SceneWeaver instead of the strongest baseline | open-vocabulary scene generation · Open-vocabulary average (8 room types) | Real.: 7.7 -> 8.8 | Helped |
| use SceneWeaver instead of the strongest baseline | open-vocabulary scene generation · Open-vocabulary average (8 room types) | #CN: 0.1 -> 0.0 | Helped |
| add reflection to SceneWeaver (ablation) | SceneWeaver · Kitchen (3-scene ablation) | Comp.: 6.3 -> 7.7 | Helped |
| add physical optimization to SceneWeaver (ablation) | SceneWeaver · Kitchen (3-scene ablation) | #CN: 2.0 -> 0.0 | Helped |
| use iterative reflection instead of single-shot multi-step planning (ablation) | SceneWeaver · Kitchen (3-scene ablation) | Func.: 7.7 -> 9.3 | Helped |
| add modifier tools to initializer tools (ablation) | SceneWeaver · Kitchen (3-scene ablation) | Func.: 7.0 -> 8.3 | Helped |
| add modifier tools to initializer tools (ablation) | SceneWeaver · Kitchen (3-scene ablation) | #Obj: 23.0 -> 16.3 | Hurt |
| add modifier tools to initializer tools (ablation) | SceneWeaver · Kitchen (3-scene ablation) | Comp.: 5.7 -> 5.0 | Hurt |
| add implementer tools to initializer tools (ablation) | SceneWeaver · Kitchen (3-scene ablation) | Comp.: 5.7 -> 7.3 | Helped |
| combine initializer, implementer, and modifier tools (ablation) | SceneWeaver · Kitchen (3-scene ablation) | Real.: 8.0 -> 9.0 | Helped |
| limit SceneWeaver to 3 iterations | SceneWeaver · Bedroom | GPT score: 8.6 | No clear effect |
| use SceneWeaver instead of baseline scene generators | human evaluation · Open-vocabulary scenes | Real.: 6.70 -> 8.80 | Helped |
Surprising findings
- Data-driven methods outperform LayoutGPT on physical metrics, suggesting LLM-only generation is insufficient for physical plausibility.
- Holodeck's higher object count is attributed to randomly placed objects lacking rational placement.
Limitations
- Objaverse assets lack standard sizes and front directions, leading to occasional unrealistic scaling and orientation for functional items.
- Relies on GPT-4 both for generation and for evaluation, raising concerns about judge validity and circularity.
- Long generation time and API cost per scene (up to 130 minutes; several tool types are slow).
Experimental setup
Datasets: 3D-FRONT 3D-FUTURE Objaverse Infinigen assets MetaScenes
Benchmarks: common room types (bedroom, living room) open-vocabulary room types (8 types)
Metrics: #Obj #OB #CN Real. Func. Lay. Comp. shift distance in Isaac Sim human preference and diversity
Baselines: ATISS DiffuScene PhyScene LayoutGPT Holodeck I-Design
Builds on: ReAct OpenManus Infinigen Holodeck ATISS
Closest papers by meaning
- InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior (ICLR 2024 · similarity 0.91)
- SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code (ICML 2024 · similarity 0.90)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.90)
- SceneCraft: Layout-Guided 3D Scene Generation (NeurIPS 2024 · similarity 0.89)
- PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI (CVPR 2024 · similarity 0.89)
- Video Perception Models for 3D Scene Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.