Research map / Image generation / Text-to-image diffusion
ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConceptMix, an automatic compositional text-to-image benchmark that samples k visual concepts, uses GPT-4o to write prompts, and grades images with GPT-4o question answering.
Problem
Existing compositional text-to-image benchmarks rely on fixed templates with few concepts per prompt, giving low diversity and weak discriminative power for evaluating T2I models.
Approach
Randomly sample one object plus k visual concepts from eight categories, have GPT-4o bind them into a JSON structure and write a prompt, then grade each image by answering one yes/no question per concept with GPT-4o, scoring full-mark and concept-fraction metrics.
Key results
- DALL·E 3 full-mark score falls from 0.83 at k=1 to 0.50 at k=3 and 0.08 at k=7, the best of the eight evaluated models.
- SD v1.4 drops to 0.08 at k=3 and 0.00 at k=6, the steepest decline among the models tested.
- GPT-4o grading agrees with the human majority vote at 0.81 consistency, comparable to the 0.85 inter-annotator consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of added visual concepts from k=1 to k=7 | DALL·E 3 · ConceptMix | full-mark score: 0.83 -> 0.08 | Hurt |
| Increase the number of added visual concepts from k=1 to k=7 | Playground v2.5 · ConceptMix | full-mark score: 0.70 -> 0.00 | Hurt |
| Increase the number of added visual concepts from k=1 to k=7 | SDXL Base · ConceptMix | full-mark score: 0.69 -> 0.00 | Hurt |
| Increase the number of added visual concepts from k=1 to k=7 | DeepFloyd IF XL v1 · ConceptMix | full-mark score: 0.68 -> 0.01 | Hurt |
| Increase the number of added visual concepts from k=1 to k=7 | PixArt alpha · ConceptMix | full-mark score: 0.66 -> 0.01 | Hurt |
| Use easy concepts rather than the standard concept sets | T2I models on ConceptMix · ConceptMix easy variant | full-mark score | Helped |
| Use hard concepts rather than the standard concept sets | T2I models on ConceptMix · ConceptMix hard variant | full-mark score | Hurt |
| Use concept-by-concept GPT-4o grading instead of T2VScore | T2I model evaluation · ConceptMix across k | alignment with human preferences | Helped |
Surprising findings
- LAION-5B captions average only 2.75 concept categories per example, with shape and spatial concepts rarely present, suggesting training data lacks complex concept combinations.
- T2VScore barely changes as k increases and does not correlate well with human scores, unlike the ConceptMix grading.
Limitations
- Automatic GPT-4o grading may miss nuances human graders capture, especially for ambiguous images.
- Human evaluators disagreed substantially on complex prompts, and GPT-4o was stricter in some cases.
- Prompt rejection removes 13-52% of initially generated prompts, mainly for shape conflicts.
Experimental setup
Datasets: LAION-5B
Benchmarks: ConceptMix
Metrics: full-mark score concept fraction score consistency with human majority vote
Baselines: T2I-CompBench T2VScore CC-500 ABC-6K Attn-Exct HRS-comp
Builds on: T2I-CompBench SkillMix
Closest papers by meaning
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.91)
- SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis (NeurIPS 2026 · similarity 0.89)
- Progressive Compositionality in Text-to-Image Generative Models (ICLR 2025 · similarity 0.89)
- Text-to-Image Generation for Abstract Concepts (AAAI 2024 · similarity 0.89)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.89)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.