Research map / Image generation / Text-to-image diffusion
Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Gecko, an evaluation suite (Gecko2K prompts, ~100K human annotations across four templates) for text-to-image alignment, plus a QA/VQA auto-eval metric that improves on prior interpretable metrics across point-wise, pair-wise, and model-ordering tasks.
Problem
Evaluating text-to-image alignment is unreliable because results depend on the prompt set, human annotation template, and evaluation task; single-slice evaluations give conclusions that do not generalise.
Approach
Systematically combine a skill-based prompt set (Gecko(S), LLM-generated with sub-skills) and a resampled DSG1K set (Gecko(R)), collect ratings under multiple templates (Likert, Word Level, DSG(H), side-by-side), and evaluate metrics on three tasks; the Gecko metric adds coverage-grounded QA generation, NLI-based hallucination filtering, and normalized VQA scores to a TIFA-style pipeline.
Key results
- Gecko metric with Gemini Flash reaches Spearman 0.58 on Gecko(R) and 0.59 on Gecko(S) for Likert ratings, outperforming DSG and TIFA with the same PaLM-2/PaLI backend.
- Gecko with Gemini Flash reaches 78.8% SxS accuracy on Gecko(S), above VQAScore (76.5%) and CLIP-based baselines.
- Model rankings change with prompt set and template; e.g., SDXL appears best on Gecko(R) by average ratings but Muse and SDXL are not significantly different under the Wilcoxon test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gecko instead of TIFA with the same PaLM-2/PaLI backend | T2I alignment metric · Gecko2K, across annotation templates and tasks | correlation with human ratings / SxS accuracy | Helped |
| Replace the PaLM-2/PaLI backend with Gemini Flash for Gecko | Gecko metric · Gecko(R) and Gecko(S) | SpearmanR and SxS accuracy: 0.41 -> 0.43; 0.55 -> 0.58; 0.46 -> 0.48; 62.1 -> 72.2; 0.47 -> 0.54; 0.52 -> 0.59; 0.45 -> 0.56; 74.6 -> 78.8 | Helped |
| Use reliable prompts rather than the full prompt set | Human SxS annotations · Gecko(R) and Gecko(S) | Krippendorff's alpha: 0.45 to 0.47; 0.49 to 0.54 | Helped |
| Add coverage-guided question generation to TIFA (ablation) | Gecko metric · Gecko2K | correlation with human judgement | Helped |
| Add VQA score linear normalisation to TIFA (ablation) | Gecko metric · Gecko2K | correlation with human judgement | Helped |
| Add NLI filtering of hallucinated QA pairs to TIFA (ablation) | Gecko metric · Gecko2K | correlation with human judgement | Helped |
| Use fine-grained Word Level and DSG(H) templates instead of Likert | Human annotation · Gecko2K | inter-annotator agreement | Helped |
Surprising findings
- Average-rating model rankings can disagree with significance-tested rankings, so SDXL vs Muse conclusions depend on the test used.
- Single-score CLIP-based metrics like PyramidCLIP are much better on pair-wise SxS accuracy despite worse point-wise correlation.
Limitations
- Gecko(R) is limited by automatic tagging errors and does not cover all skills or sub-skills.
- Gecko2K-rel removes hard prompts, so conclusions on the reliable subset may not generalise to difficult prompts.
- The NLI filtering threshold was set by inspecting QA pairs, and the metric depends on the quality of the underlying PaLM-2/PaLI or Gemini models.
Experimental setup
Datasets: DSG1K TIFA160 MSCOCO PartiPrompts DrawBench
Benchmarks: Gecko2K TIFA160 VBench
Metrics: Spearman correlation Pearson correlation SxS accuracy Krippendorff's alpha Wilcoxon signed-rank significance
Baselines: TIFA DSG CLIP PyramidCLIP VQAScore VNLI DrawBench PartiPrompts
Builds on: TIFA DSG VQAScore DSG1K
Closest papers by meaning
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.93)
- Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.91)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.91)
- FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.90)
- Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality (NeurIPS 2024 · similarity 0.89)
- Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist Rewards (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.