Research map / Image generation / Text-to-image diffusion
ImagenHub: Standardizing the evaluation of conditional image generation models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ImagenHub, a unified library, curated datasets, and human evaluation protocol (SC and PQ scores) to standardize inference and evaluation of about 30 conditional image generation and editing models across seven tasks.
Problem
Published conditional image generation and editing models are evaluated under inconsistent datasets, inference settings, and human evaluation protocols, making fair comparison and tracking of progress difficult.
Approach
A one-stop library that fixes datasets, hyper-parameters, and prompt formats across seven conditional generation tasks, paired with a unified three-level human rating guideline (SC and PQ in {0, 0.5, 1}) combined into an overall score via geometric mean.
Key results
- Human evaluation reached Krippendorff's alpha above 0.4 on 76% of models; 74% of models scored below 0.5 overall except in text-guided and subject-driven generation.
- 83% of published claims about model rankings were found to hold under the ImagenHub evaluation.
- Automatic metrics showed Spearman correlation with human scores below 0.2 for most tasks, except subject-driven image editing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Dalle-3 rather than Midjourney | Text-guided Image Generation · Text-guided Image Generation | Overall: 0.73$\pm$0.07 -> 0.76$\pm$0.08 | Helped |
| Use SDXL-Inpainting rather than SD-Inpainting | Mask-guided Image Editing · Mask-guided Image Editing | Overall: 0.17$\pm$0.07 -> 0.37$\pm$0.05 | Helped |
| Use MagicBrush rather than InstructPix2Pix | Text-guided Image Editing · Text-guided Image Editing | Overall: 0.27$\pm$0.02 -> 0.47$\pm$0.02 | Helped |
| Use SuTI rather than DreamBooth | Subject-driven Image Generation · Subject-driven Image Generation | Overall: 0.55$\pm$0.11 -> 0.58$\pm$0.12 | Helped |
| Use PhotoSwap rather than DreamEdit | Subject-driven Image Editing · Subject-driven Image Editing | Overall: 0.32$\pm$0.03 -> 0.36$\pm$0.02 | Helped |
| Use CustomDiffusion rather than DreamBooth | Multi-concept Image Composition · Multi-concept Image Composition | Overall: 0.13$\pm$0.02 -> 0.29$\pm$0.01 | Helped |
| Switch from ControlNet to UniControl | Control-guided Image Generation · Control-guided Image Generation | Overall: 0.23$\pm$0.04 vs 0.23$\pm$0.07 | No clear effect |
| Replace geometric-mean scoring with weighted sum O = 0.5SC + 0.5PQ (ablation) | ImagenHub overall human score · Text-guided Image Editing | Overall average score: CycleDiffusion: 0.14 -> 0.36; Text2Live: 0.02 -> 0.42 | Mixed |
| Replace geometric-mean scoring with weighted sum O = 0.7SC + 0.3PQ (ablation) | ImagenHub overall human score · Text-guided Image Editing | Overall average score: CycleDiffusion: 0.14 -> 0.29; Text2Live: 0.02 -> 0.26 | Mixed |
| Expand SC and PQ categories from [0, 0.5, 1] to [0, 0.5, 1, 2] (ablation) | ImagenHub human evaluation · Subject-driven Image Editing | Fleiss' Kappa: 0.35 -> 0.25 | Hurt |
| Narrow SC and PQ categories from [0, 0.5, 1] to [0, 1] (ablation) | ImagenHub human evaluation · Subject-driven Image Editing | Fleiss' Kappa: 0.35 -> 0.37 | Mixed |
Surprising findings
- Automatic metrics such as CLIP score and LPIPS correlate poorly with human judgments in most tasks.
- Text2Live often returns the input image unchanged, which the weighted-sum overall score fails to penalize, motivating the geometric mean.
Limitations
- Relies on human raters, which is expensive and time-consuming.
- Automatic metrics do not correlate well with human judgments, so no cheap proxy is available.
- Datasets are small (100-200 instances per task) and the leaderboard requires continuous manual evaluation of new models.
Experimental setup
Datasets: DrawBench DiffusionDB ABC-6K MagicBrush SuTI CustomDiffusion DreamEditBench
Benchmarks: ImagenHub dataset
Metrics: Semantic Consistency Perceptual Quality overall score (geometric mean) Fleiss kappa Krippendorff's alpha LPIPS CLIP score DINO CLIP-I DreamSim
Baselines: Dalle-3 Midjourney Stable Diffusion XL MagicBrush InstructPix2Pix DreamBooth ControlNet UniControl
Builds on: ControlNet DreamBooth Textual Inversion CLIP Stable Diffusion
Closest papers by meaning
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.89)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.89)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.89)
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.88)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.88)
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.