Research map / Image generation / Text-to-image diffusion
GenEval: An object-focused framework for evaluating text-to-image alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add background masking and bounding-box cropping before CLIP color classification (ablation) | GenEval color classifier · attribute binding | Cohen's kappa: 0.01 -> 0.49 | Helped |
| add crop and mask for colors task (ablation) | GenEval color classifier · colors | Cohen's kappa: 0.32 -> 0.45 | Helped |
| raise detector confidence threshold from 0.3 to 0.9 for counting (ablation) | GenEval · counting | Cohen's kappa: 0.37 -> 0.65 | Helped |
| add a minimum offset distance (c=0.1) for position classification (ablation) | GenEval · position | human agreement | Helped |
| add non-maximal suppression to detector outputs (ablation) | GenEval · counting | human agreement | No clear effect |
| replace CLIPScore with GenEval as evaluator | CLIPScore ViT-H/14 · overall human agreement | agreement: 0.798 -> 0.834 | Helped |
| replace CLIPScore with GenEval | CLIPScore ViT-H/14 · counting | agreement: 0.734 -> 0.823 | Helped |
| replace CLIPScore with GenEval | CLIPScore ViT-H/14 · single object and colors | agreement: colors 0.827 -> 0.808 | Hurt |
| scale IF model from M to XL (ablation) | IF · GenEval overall | GenEval score: 0.52 -> 0.61 | Helped |
| scale IF model from M to XL (ablation) | IF · position | GenEval score: 0.10 -> 0.13 | No clear effect |
| increase SDv1 pretraining steps from v1.2 to v1.5 (ablation) | Stable Diffusion v1 · GenEval overall | GenEval score: 0.44 -> 0.43 | No clear effect |
| switch to SDv2 with OpenCLIP ViT-H/14 text encoder | Stable Diffusion · GenEval overall | GenEval score: 0.43 -> 0.51 | Helped |
Closest papers by meaning
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.89)
- Re-Aligning Language to Visual Objects with an Agentic Workflow (ICLR 2025 · similarity 0.88)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
- FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.