Research map / Image generation / Text-to-image diffusion
Holistic Evaluation of Text-to-Image Models
Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park and 6 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Promptist prompt engineering to Stable Diffusion v1-4 | Stable Diffusion v1-4 · HEIM aesthetics | human-rated aesthetics | Improved |
| Apply Promptist prompt engineering to Stable Diffusion v1-4 | Stable Diffusion v1-4 · HEIM alignment | text-image alignment | No clear effect |
| Fine-tune on artistic images | Stable Diffusion-based models · HEIM aesthetics | aesthetics | Improved |
| Fine-tune on artistic images | Stable Diffusion-based models · HEIM photorealism | human-rated photorealism | Hurt |
| Fine-tune on custom data | Text-to-image models · MS-COCO gender and dialect perturbations | human-rated alignment gap | Hurt |
| Increase SafeStableDiffusion safety guidance strength | SafeStableDiffusion · I2P | inappropriate-image generation frequency | Improved |
| Use safety guidance | SafeStableDiffusion · HEIM alignment | human-rated text-image alignment | Hurt |
| Increase model size (ablation) | DALL-E and DeepFloyd-IF model families · MS-COCO | human metrics | Improved |
| Translate MS-COCO prompts into Hindi, Chinese, or Spanish | Most evaluated text-to-image models · MS-COCO translated prompts | text-image alignment | Hurt |
| Translate MS-COCO prompts into Chinese | CogView2 · MS-COCO Chinese prompts | text-image alignment | Improved |
| Introduce typos in prompts | Text-to-image models · MS-COCO robustness | human-rated alignment: no more than 0.2 | No clear effect |
| Use diffusion rather than autoregressive generation at similar parameter count | Text-to-image models · HEIM efficiency | inference efficiency | Improved |
Closest papers by meaning
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.90)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.89)
- Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content (CVPR 2025 · similarity 0.89)
- ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR 2024 · similarity 0.89)
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.87)
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
