Research map / Image generation / Text-to-image diffusion
Muses: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUSES, a multi-agent pipeline (Layout Manager, Model Engineer, Image Artist) that lifts 2D LLM layouts to 3D, retrieves or generates 3D models, aligns them with a fine-tuned CLIP, renders depth/edge conditions in Blender, and generates images with SD3 ControlNet, plus a new 50-prompt benchmark T2I-3DisBench.
Problem
Text-to-image models struggle to control 3D properties such as object orientation, 3D spatial relationships, and camera view when prompts contain multiple objects, and existing benchmarks lack detailed 3D prompts.
Approach
Mimic 3D professional workflows with a multi-agent system: an LLM plans a 2D layout via in-context learning and lifts it to 3D via chain-of-thought, 3D models are retrieved or generated and face-camera aligned with a fine-tuned CLIP, and Blender renders depth and edge conditions for ControlNet-based generation.
Key results
- On T2I-CompBench, MUSES reaches 0.8726 color, 0.4756 2D-spatial, and 0.7720 numeracy, exceeding SD v3 (0.8085, 0.3144, 0.6088).
- On T2I-3DisBench, MUSES attains an average InternVL-VQA score of 0.6156 versus 0.4206 for SD v3 and 0.3760 for Midjourney v6.0.
- User evaluation with 30 participants on 20 prompts also favors MUSES across the four 3D dimensions (average 0.62 vs 0.36 for SD v3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Object Depth Planning (ablation) | MUSES · T2I-CompBench | 3D-Spatial: 0.4127 -> 0.4639 | Helped |
| add Object Depth Planning (ablation) | MUSES · T2I-3DisBench | Average InternVL-VQA score: 0.4981 -> 0.6156 | Helped |
| add Object Orientation Planning (ablation) | MUSES · T2I-CompBench | 3D-Spatial: 0.4609 -> 0.4639 | Helped |
| add Object Orientation Planning (ablation) | MUSES · T2I-3DisBench | Average InternVL-VQA score: 0.4938 -> 0.6156 | Helped |
| add Camera View Planning (ablation) | MUSES · T2I-3DisBench | Average InternVL-VQA score: 0.5245 -> 0.6156 | Helped |
| add Retrieve-Generate Decision Tree (ablation) | MUSES · T2I-CompBench | 3D-Spatial: 0.4234 -> 0.4639 | Helped |
| add Face-Camera Calibration (ablation) | MUSES · T2I-3DisBench | Average InternVL-VQA score: 0.4019 -> 0.6156 | Helped |
| fine-tune CLIP during calibration (ablation) | MUSES · T2I-3DisBench | Average InternVL-VQA score: 0.5026 -> 0.6156 | Helped |
| use Multiple Control Scales (ablation) | MUSES · T2I-CompBench | 3D-Spatial: 0.3893 -> 0.4639 | Helped |
| add 3D Layout Shop Expansion (ablation) | MUSES · T2I-3DisBench | Average InternVL-VQA score: 0.4405 -> 0.6156 | Helped |
| increase ControlNet control scale (ablation) | MUSES · image-condition alignment | alignment | Helped |
| increase ControlNet control scale (ablation) | MUSES · generated images | image fidelity | Hurt |
Surprising findings
- Removing the Face-Camera Calibration module causes the sharpest performance drop, indicating orientation alignment matters more than other components.
- The authors note that MUSES is less efficient than SD3 due to multi-agent overhead and fails on prompts with many objects and unclear layouts (e.g., 'twenty five bread').
Limitations
- Lower efficiency than SD3 due to multi-agent pipeline.
- Fails on prompts with many objects and unclear layout descriptions.
- Relies on a small self-built 3D model shop (300 models) and CLIP-based online model matching, with lighting control not addressed.
Experimental setup
Datasets: NSR-1K T2I-CompBench CGTrader online 3D models (self-collected 3D shop of 300 models)
Benchmarks: T2I-CompBench T2I-3DisBench
Metrics: T2I-CompBench attribute binding, object relationship, numera InternVL-VQA score on count, orientation, 3D spatial relatio user rating (0-1) on four dimensions
Baselines: LayoutGPT Structured Diffusion Attn-Exct GORS RPG-Diffusion CompAgent SDXL PixArt-alpha Playground v2.5 Hunyuan-DiT DALL-E 3 Stable Diffusion v3
Builds on: ControlNet CLIP LayoutGPT RPG-Diffusion CompAgent
Closest papers by meaning
- SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent (NeurIPS 2025 · similarity 0.88)
- ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary (CVPR 2025 · similarity 0.88)
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.88)
- ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling (NeurIPS 2025 · similarity 0.87)
- 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.87)
- Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.