Research map / Image generation / Text-to-image diffusion
GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenArtist, an MLLM-agent-coordinated system that unifies image generation and editing by decomposing prompts, planning a verified tool tree with self-correction, and adding position-aware tool inputs.
Problem
Single image generation or editing models struggle with complex prompts, lack verification and self-correction, and specialize in narrow tasks, so no single model satisfies diverse user requirements for generation and editing.
Approach
An MLLM agent (GPT-4V) decomposes complex requests into simple sub-operations, organizes candidate tools (generation and editing models) into a planning tree with per-node verification and editing-based self-correction subtrees, and uses an object detector plus auxiliary localization tools to supply position information for tool calls.
Key results
- On T2I-CompBench, GenArtist reaches 0.8482 color, 0.5437 spatial and 0.4499 complex, beating DALL-E 3 (0.7785 color, 0.2865 spatial, 0.3773 complex).
- On MagicBrush single-turn editing, GenArtist attains L1 0.0536 and CLIP-T 0.3129, outperforming MagicBrush (L1 0.0625, CLIP-T 0.2781).
- Ablation on T2I-CompBench shows the planning tree improves over the planning chain and tool selection alone (e.g., spatial 0.4787 to 0.5437), and removing position information drops spatial from 0.5437 to 0.4577.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenArtist instead of DALL-E 3 | T2I image generation · T2I-CompBench | Color: 0.7785 -> 0.8482 | Helped |
| use GenArtist instead of DALL-E 3 | T2I image generation · T2I-CompBench | Spatial: 0.2865 -> 0.5437 | Helped |
| use GenArtist instead of MagicBrush | single-turn image editing · MagicBrush single-turn | CLIP-T: 0.2781 -> 0.3129 | Helped |
| use GenArtist instead of MagicBrush | multi-turn image editing · MagicBrush multi-turn | L1: 0.0964 -> 0.0858 | Helped |
| add MLLM tool selection (ablation) | GenArtist · T2I-CompBench | Spatial: 0.2064 -> 0.4305 | Helped |
| add planning chain for verification and correction (ablation) | GenArtist with tool selection · T2I-CompBench | Complex: 0.3739 -> 0.4095 | Helped |
| replace planning chain with planning tree (ablation) | GenArtist with planning chain · T2I-CompBench | Spatial: 0.4787 -> 0.5437 | Helped |
| add position information (ablation) | GenArtist tool selection · T2I-CompBench | Spatial: 0.4577 -> 0.5437 | Helped |
| add position information (ablation) | GenArtist tool selection · T2I-CompBench | Complex: 0.4083 -> 0.4499 | Helped |
Surprising findings
- Position information from an object detector lifts spatial accuracy substantially (0.4577 to 0.5437), suggesting MLLMs are position-insensitive without it.
Limitations
- Performance depends on the underlying MLLM agent (GPT-4V); tasks beyond its capability may fail.
- Errors from underlying editing tools (e.g., fine-resolution small objects) and localization tools (segmentation errors) propagate to final output.
- Potential misuse of image generation and editing raises negative social impact concerns.
Experimental setup
Datasets: T2I-CompBench MagicBrush
Benchmarks: T2I-CompBench MagicBrush
Metrics: Color Shape Texture Spatial Non-Spatial Complex L1 L2 CLIP-I DINO
Baselines: SDXL DALL-E 3 PixArt-alpha CompAgent RPG Null Text Inversion HIVE InstructPix2Pix MagicBrush SmartEdit
Builds on: LMD CompAgent ControlNet DALL-E 3
Closest papers by meaning
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.87)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.87)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.86)
- OmniGen2: Towards Instruction-Aligned Multimodal Generation (CVPR 2026 · similarity 0.86)
- The Promise of RL for Autoregressive Image Editing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.