Research map / Image generation / Text-to-image diffusion
ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ColorConceptBench, a human-grounded benchmark of 1,281 implicit color concepts (visual states and emotions) with designer colorizations, and uses distribution-based metrics (PCC, EMD, entropy difference) to evaluate nine open T2I models.
Problem
Text-to-image models are evaluated for color alignment only with explicit color names or codes and deterministic single-point references, which ignores implicit color-concept associations (emotions, visual states) that humans hold as probabilistic distributions.
Approach
Builds human-grounded color distributions from professional designers coloring sketches for each concept, extracts CIELAB color distributions via segmentation and adaptive grouping, and compares model-generated distributions to these using distribution metrics (EMD, PCC, ED) rather than single dominant-color checks.
Key results
- Sana-1.5 achieves the best overall probabilistic alignment, with the best PCC on Original Clipart (0.428) and EMD on Original Clipart (26.151).
- Models show under-sensitivity to modifiers: average color shift is about 77% of human shift for Visual State and about 72% for Emotional modifiers.
- EMD correlates best with human pairwise judgments (Kendall's Tau 0.276, agreement 68.37%) versus deterministic DCA (agreement 11.94%).
Surprising findings
- Increasing classifier-free guidance generally worsens EMD and does not improve color-concept alignment, with OmniGen degrading as guidance grows.
- Alignment is higher for Clipart than Natural style, and for original concepts than for visual state or emotional variants.
Limitations
- Human annotations come predominantly from a single cultural region, so color associations may not generalize.
- Base concepts are restricted to tangible objects; culturally specific cues are omitted.
- Emotional modifiers are restricted to facility and landscape entities.
Experimental setup
Datasets: THINGS COCA word frequency
Benchmarks: ColorConceptBench
Metrics: PCC EMD Entropy Difference DCA Hue Angular Difference Kendall's Tau Spearman correlation agreement ratio
Baselines: Flux.1-dev Qwen-Image OmniGen OmniGen2 PixArt-alpha SD 3 SD 3.5 SD XL Sana-1.5
Builds on: THINGS dataset Grounding DINO Segment Anything Model CLIPScore UW71 color space
Closest papers by meaning
- MegaCoin: Enhancing Medium-Grained Color Perception for Vision-Language Models (AAAI 2026 · similarity 0.87)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.87)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.85)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.85)
- UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.