Research map / 3D vision / 3D generation
Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Architect generates interactive 3D indoor and non-household scenes by hierarchically inpainting simulation-rendered images with a 2D diffusion model, then lifting them to 3D with depth estimation and retrieved or generated assets.
Problem
Generating diverse, realistic, and complex interactive 3D embodied environments is hard: manual design and procedural rules don't scale, dataset-trained diffusion models are bounded by their training data, and LLMs have weak 3D spatial reasoning.
Approach
Use pre-trained 2D inpainting diffusion models as layout priors, controlled by inpainting over a simulator-rendered background image so camera parameters and depth scale are known; iteratively inpaint large furniture and then small objects, perception-lift the results to 3D bounding boxes, and place retrieved or generated assets.
Key results
- Architect achieves the highest CLIP score (0.7173) and BLIP score (0.5859) among compared methods, and the best GPT-4o ranking (1.36).
- In a 115-participant user study, Architect scores highest on visual quality (3.87), semantic correctness (3.76), layout correctness (3.65) and overall preference (3.71).
- Against PhyScene, Architect reaches CLIP 72.96, BLIP 63.62 and VQScore 94.58 on living-room generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical inpainting and 3D scene placement instead of LLM placement | Small object placement · Small object placement | VQScore: 74.93 -> 80.81 | Helped |
| Use Architect's scene-generation pipeline instead of RoboGen | Task description generation · Task description diversity | Self-BLEU: 0.284 -> 0.198 | Helped |
| Soften inpainting-mask boundaries with blur | Architect inpainting · Inpainting consistency | visual consistency | Helped |
| Use Architect rather than the strongest reported baseline | Architect · Scene generation evaluation | CLIP: 0.6785 -> 0.7173 | Helped |
| Use Architect rather than the strongest reported baseline | Architect · Scene generation evaluation | BLIP: 0.1223 -> 0.5859 | Helped |
| Use Architect rather than Text2Room | Architect · Scene generation evaluation | VQA: 0.8149 -> 0.8073 | Hurt |
| Use Architect rather than Holodeck | Architect · Living room generation | CLIP: 71.42 -> 72.96 | Helped |
| Use Architect rather than Holodeck | Architect · Living room generation | BLIP: 53.23 -> 63.62 | Helped |
| Use Architect rather than Text2Room | Architect · Living room generation | VQScore: 90.73 -> 94.58 | Helped |
Surprising findings
- Text2Room achieves the highest VQA score (0.8149) despite its much lower visual quality in the user study.
Limitations
- Furniture and large objects are retrieved from datasets, so furniture diversity is bounded by the asset database.
- Scene diversity is constrained by the retrieved asset library; articulated objects are not generated.
- Including many small objects slows the simulation, requiring LLM-based selection of physically simulated objects.
Experimental setup
Datasets: Objaverse PartNet-Mobility
Benchmarks: CLIPScore BLIPScore VQAScore GPT-4o ranking user study
Metrics: CLIP score BLIP score VQA score GPT-4o ranking visual quality semantic correctness layout correctness overall preference self-BLEU
Baselines: Holodeck Text2Room DiffuScene PhyScene
Builds on: Holodeck Latent Diffusion Marigold RoboGen DreamFusion
Closest papers by meaning
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.89)
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.88)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.88)
- SceneCraft: Layout-Guided 3D Scene Generation (NeurIPS 2024 · similarity 0.88)
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning (NeurIPS 2025 · similarity 0.87)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.