Research map / Image generation / Text-to-image diffusion
IDEA-Bench: How Far are Generative Models from Professional Designing?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IDEA-Bench, a benchmark of 100 professional image-generation tasks (275 cases) across five task types, scored with six hierarchical binary questions by humans and with Gemini 1.5 Pro on an 18-task mini subset.
Problem
Existing image generation and editing benchmarks are narrow and academic, and do not measure the multi-image, complex-instruction demands of professional designers.
Approach
A professional-task benchmark organized into five capability-based categories (text-to-image, image-to-image, images-to-image, text-to-images, image(s)-to-images), with hierarchical binary evaluation questions where higher-level quality scores are zeroed unless task requirements are met, and MLLM (Gemini 1.5 Pro) scoring validated against humans.
Key results
- FLUX-1 with GPT-4o prompt rephrasing achieves the highest overall score of 22.48/100, while no model without MLLM assistance exceeds 10 overall.
- Emu2 attains the top images-to-image category score of 8.94 and the top non-MLLM result on IDEA-Bench-mini.
- Anole scores 0.48/100 overall, and models generally struggle with text generation, information charts, and consistent multi-image generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o to rephrase prompts for FLUX-1 | FLUX-1 · IDEA-Bench | average score: 6.81 -> 22.48 | Helped |
| Use GPT-4o to rephrase prompts for OmniGen | OmniGen · IDEA-Bench | average score: 6.47 -> 15.45 | Helped |
| Use GPT-4o to rephrase prompts for Emu2 | Emu2 · IDEA-Bench | average score: 6.81 -> 12.46 | Helped |
| Use GPT-4o to rephrase prompts for SD3 | SD3 · IDEA-Bench | average score: 14.83 -> 14.83 | Mixed |
| Use GPT-4o to rephrase prompts for DALL-E 3 | DALL-E 3 · IDEA-Bench | average score: 13.07 -> 13.07 | Mixed |
| Use GPT-4o to rephrase prompts for Pixart | Pixart · IDEA-Bench | average score: 12.90 -> 12.90 | Mixed |
| Use an image-editing model rather than a general-purpose model | Image-to-Image tasks · Image-to-Image | average score: 19.07 | Helped |
| Use Emu2 for multi-image-guided image generation | Image-to-Image tasks · Images-to-Image | average score: 8.98 | Helped |
| Use an image-editing model for local editing | Image editing models · Image-to-Image | task completion | Helped |
| Use an image-editing model for subject-driven tasks | Image editing models · Image-to-Image | task completion | Hurt |
| Generate text within images | Text-to-Image models · Text-to-Image | task completion | Hurt |
| Generate information charts with current image-generation models | Text-to-Image models · Text-to-Image | task completion | Hurt |
Surprising findings
- Without MLLM prompt rephrasing, no model surpasses a score of 10 overall, indicating general image generators fall short of professional-grade performance.
- Specialized image editing models (MagicBrush, InstructPix2Pix) rank top on image-to-image tasks but fail at subject-driven tasks like branded merchandise generation.
Limitations
- MLLM judge (Gemini 1.5 Pro) still struggles to match humans on complex tasks and is sensitive to image order, requiring manual question refinement.
- Several baselines rely on GPT-4o prompt rephrasing, so scores mix model capability with MLLM planning.
Experimental setup
Datasets: IDEA-Bench
Benchmarks: IDEA-Bench IDEA-Bench-mini
Metrics: hierarchical binary score CLIPScore average task score
Baselines: FLUX-1 DALL-E 3 Stable Diffusion 3 Pixart InstructPix2Pix MagicBrush Anole Emu2 OmniGen
Builds on: DALL-E 3 ImagenHub GenEval DrawBench
Closest papers by meaning
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning (ICLR 2026 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.88)
- KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models (NeurIPS 2025 · similarity 0.88)
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.