Research map / Vision-language and multimodal / Human and embodied modeling
Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces T2IScoreScore (TS2), a benchmark of semantic error graphs with three meta-metrics (rank, separation, delta) for objectively scoring T2I prompt-faithfulness metrics, finding cheap CLIPScore competitive with VLM-based metrics.
Problem
Existing T2I prompt-faithfulness metrics are validated ad hoc against noisy human Likert preferences on small, low image-per-prompt sets, so they cannot be rigorously compared.
Approach
Organize images for each prompt into semantic error graphs (SEGs) whose nodes have objectively known error counts, and score metrics by whether they order images along error walks (Spearman rho) and statistically separate adjacent error nodes (KS statistic and normalized mean delta).
Key results
- CLIPScore achieves average rank score 71.4% and sep 90.7%, with an estimated cost of about 604M FLOP per image, and lies on the Pareto frontier with DSG/TIFA using GPT-4V.
- The best DSG variant (BLIP/InstructBLIP-based, caption-based metrics aside) reaches rank 79.0% average, only modestly above CLIPScore's 71.4%, while VLM-based metrics cost roughly 6 orders of magnitude more compute.
- Across 165 SEGs and roughly 93,000 pairwise image comparisons, the Real subset is hardest; QG/A metrics like TIFA and DSG with GPT-4V reach about 69-70% rank on Real.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cheap embedding-correlation metric (CLIPScore) instead of VLM-based QG/A or caption metrics | T2I faithfulness metrics · TS2 all SEGs | rank_m and sep_m | No clear effect |
| Use GPT4-V as VQA backend for DSG/TIFA instead of open VLMs (mPLUG, LLaVA, etc.) | DSG and TIFA · TS2 Real SEGs | rank_m: sub-1% gain over mPLUG on Real | Mixed |
| Switch DSG accumulation to TIFA accumulation (ablation) | DSG with mPLUG · TS2 all SEGs | rank_m: 70.4 vs 68.8 | Helped |
| Switch DSG accumulation to DSG-style accumulation (graph-aware) (ablation) | DSG with Fuyu · TS2 all SEGs | rank_m: 35.8 vs 29.5 | Helped |
| Switch DSG accumulation to TIFA accumulation (ablation) | DSG with LLaVA, InstructBLIP, BLIP1 · TS2 all SEGs | rank_m: e.g. InstructBLIP 79.0 vs 80.2 | Hurt |
| Switch DSG accumulation to TIFA accumulation (ablation) | DSG/TIFA scoring with multiple VLM backends · TS2 all SEGs | sep_m | Mixed |
Surprising findings
- Simple embedding-based CLIPScore and ALIGNScore are Pareto-optimal against expensive VLM-based metrics such as TIFA and DSG with GPT-4V.
- Going from mPLUG-based to GPT-4V-based VLM backends yielded under 1% rank improvement on the hardest Real subset.
Limitations
- Modest number of prompts (165 SEGs) due to the high image-to-prompt ratio requirement.
- Rank correlation penalizes embedding metrics with continuous scores relative to discrete VLM scores due to ties.
- Compute cost of GPT-3/GPT-4-based metrics is only a rough third-party estimate.
Experimental setup
Datasets: MS-COCO PartiPrompts Pexels
Benchmarks: T2IScoreScore (TS2)
Metrics: Spearman rank correlation (rank_m) Kolmogorov-Smirnov separation (sep_m) normalized mean delta (delta_m) FLOPs per image evaluation
Baselines: CLIPScore ALIGNScore TIFA DSG LLMScore VIEScore
Builds on: TIFA DSG CLIPScore LLMScore VIEScore
Closest papers by meaning
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.91)
- FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.91)
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.91)
- SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis (NeurIPS 2026 · similarity 0.91)
- Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist Rewards (NeurIPS 2026 · similarity 0.90)
- OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.