Research map / Image generation / Text-to-image diffusion
OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OneIG-Bench, a multi-dimensional benchmark (alignment, text rendering, reasoning, style, diversity) with bilingual prompts and automated metrics, and evaluates about 20 text-to-image models.
Problem
Existing text-to-image benchmarks cover only narrow dimensions (mostly prompt alignment) and lack reliable evaluation of text rendering, style, reasoning-driven generation, and diversity.
Approach
A six-category prompt suite (General Object, Portrait, Anime and Stylization, Text Rendering, Knowledge and Reasoning, Multilingualism) with per-dimension automated metrics: DSG-style question graphs for alignment, Levenshtein-based text scores, reference-image style similarity, LLM2CLIP reasoning alignment, and DreamSim diversity; users can evaluate on a single dimension subset.
Key results
- GPT-4o achieves the top style score (0.462) and reasoning score (0.345) on OneIG-Bench-EN, with alignment 0.851 and text 0.857.
- Seedream 3.0 achieves the best text score on OneIG-Bench-EN (0.865) and OneIG-Bench-ZH (0.928).
- Janus-Pro and Stable Diffusion 1.5 obtain the highest diversity scores (0.365 and 0.429) but low alignment, showing diversity can reflect poor prompt adherence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chain-of-thought prompting to BAGEL (ablation) | BAGEL · OneIG-Bench-EN | alignment: 0.769 -> 0.793 | Helped |
| Add chain-of-thought prompting to BAGEL (ablation) | BAGEL · OneIG-Bench-EN | text score: 0.244 -> 0.020 | Hurt |
| Use tag-based or phrase-based prompts instead of natural-language prompts | T2I models on OneIG-Bench-EN · OneIG-Bench-EN alignment | alignment | Hurt |
| Increase prompt length | T2I models on OneIG-Bench-EN · OneIG-Bench-EN alignment | alignment | Hurt |
| Use Chinese prompts rather than English prompts | Most evaluated T2I models · OneIG-Bench-ZH vs OneIG-Bench-EN | alignment | Hurt |
| Use Chinese text prompts | Seedream 3.0 · OneIG-Bench-ZH | text score: 0.928 | Helped |
| Use Chinese text prompts | GPT-4o · OneIG-Bench-ZH | text score: 0.650 | Hurt |
| Use longer text prompts | Seedream 3.0 · OneIG-Bench-EN text rendering | edit distance: 7.204; 20.596; 169.307 | Hurt |
| Use a layout-first strategy before text insertion | Recraft V3 · OneIG-Bench-EN long text prompts | edit distance: 74.181 | Helped |
Surprising findings
- Stable Diffusion 1.5 shows strong stylization (0.383) despite weak alignment (0.565).
- GPT-4o ranks low on text edit distance because case mismatches count as errors, despite strong visual text accuracy.
Limitations
- Knowledge and reasoning metric may not be the most rational; alignment with humans only checked informally.
- Aesthetic and body-quality assessment models show biases and limited discriminative power, so those dimensions are not fully assessed.
- Closed-source model results depend on API versions at specific dates.
Experimental setup
Datasets: OneIG-Bench prompts (self-constructed from user inputs, internet data, established datasets)
Benchmarks: OneIG-Bench OneIG-Bench-ZH
Metrics: alignment score edit distance completion rate word accuracy style similarity reasoning score diversity (DreamSim)
Baselines: Janus-Pro BLIP3-o BAGEL Show-o2 OmniGen2 Stable Diffusion 3.5 Large Flux.1-dev HiDream-I1-Full Seedream 3.0 GPT-4o Imagen4 Recraft V3
Builds on: DSG DPG-Bench GenEval CSD
Closest papers by meaning
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.91)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.91)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.91)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.91)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
- Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist Rewards (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.