Research map / 3D vision / 3D generation
ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free text-to-3D scene pipeline that generates an isometric 2D image with DALLE-3, then detects objects, lifts them to 3D assets via a single-view generator, and places them using depth and pose matching.
Problem
Text-to-3D scene generation is limited by scarce 3D data and by asset retrieval from small libraries, which restricts style diversity, layout quality and generalization to new scene categories.
Approach
Uses a generated isometric 2D image as an intermediary: detect and segment objects in it, generate each object as a 3D asset from its crop and an LLM description, and recover position, size and pose from monocular depth and feature-matching against the same image, instead of learning from 3D scenes or retrieving assets.
Key results
- Object overlapping rate is 6-10x lower than LayoutGPT (e.g., 2.19 vs 27.77 for living rooms).
- CLIP score 29.45 vs 26.73 for Holodeck; GPT-4o overall winning rate 95.46% over Holodeck.
- User study style preference 82.96% vs 17.03% for Holodeck; average winning rate reported as 74.89% in abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ArtiScene instead of LayoutGPT for layout generation | Bedroom scene layout · Bedrooms | OOR: 37.26 -> 6.48 | Helped |
| Use ArtiScene instead of LayoutGPT for layout generation | Living-room scene layout · Living rooms | OOR: 27.77 -> 2.19 | Helped |
| Use ArtiScene instead of LayoutGPT for layout generation | Bedroom scene layout · Bedrooms | Avg #Furn: 4.30 -> 6.97 | Helped |
| Use ArtiScene instead of LayoutGPT for layout generation | Living-room scene layout · Living rooms | Avg #Furn: 6.23 -> 8.66 | Helped |
| Use ArtiScene instead of LayoutGPT | Bedroom scene generation · Bedrooms | Overall user preference: 27.42 -> 72.58 | Helped |
| Use ArtiScene instead of LayoutGPT | Living-room scene generation · Living rooms | Overall user preference: 12.91 -> 87.09 | Helped |
| Use ArtiScene instead of Holodeck | Text-conditioned scene generation · MIT Indoor Scenes (111 samples) | CLIP: 26.73 -> 29.45 | Helped |
| Use ArtiScene instead of Holodeck | Text-conditioned scene generation · MIT Indoor Scenes (111 samples) | Functionality: 11.84 -> 88.15 | Helped |
| Use ArtiScene instead of Holodeck | Text-conditioned scene generation · MIT Indoor Scenes (111 samples) | Style: 1.83 -> 98.16 | Helped |
| Use ArtiScene instead of Holodeck | Text-conditioned scene generation · MIT Indoor Scenes (111 samples) | Layout: 4.53 -> 95.46 | Helped |
| Use ArtiScene instead of Holodeck | Text-conditioned scene generation · MIT Indoor Scenes (39 user-study pairs) | Overall user preference: 20.26 -> 79.73 | Helped |
Limitations
- Diffusion models are weak at following detailed prompts such as object counts and locations.
- Isometric layouts are rarely used in real life, so quality drops for large-scale scenes like museums or hospitals.
- Some 3D understanding is traded for 2D data; pipeline depends on multiple model priors and the 3D generation API.
Experimental setup
Datasets: MIT Indoor Scenes
Benchmarks: LayoutGPT evaluation scenes (bedroom, living room) MIT Indoor Scenes subset (111 prompts)
Metrics: object overlapping rate average number of furniture CLIP score GPT-4o winning rate user study preference
Baselines: LayoutGPT Holodeck
Builds on: Holodeck LayoutGPT Grounding DINO Depth-Anything-2 Pix2Gestalt
Closest papers by meaning
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.89)
- Towards Text-guided 3D Scene Composition (CVPR 2024 · similarity 0.88)
- 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation (CVPR 2024 · similarity 0.88)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.88)
- Muses: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration (AAAI 2025 · similarity 0.88)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.