Research map / Vision-language and multimodal / Multimodal LLMs

FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FineGRAIN, a hierarchical framework of 27 T2I failure modes where VLMs (judged via an LLM) decide whether generated images exhibit each failure, and releases a dataset of prompts and images from 5 T2I models with human labels.

Problem

Text-to-image models often fail prompt adherence on specific attributes (object counts, color binding, text rendering), and existing T2I and VLM benchmarks are coarse and evaluate either T2I models or VLMs but not both jointly.

Approach

Pair T2I generations with VLM question-answering and an LLM judge that converts failure-mode-specific instruction templates into VLM questions and grades compliance, yielding a boolean and raw score per image across 27 fine-grained failure modes, so that both T2I models and VLM judges are evaluated on the same benchmark.

Key results

• FineGRAIN achieves 67.4% average agreement with human labels versus 57.7% for VQAScore thresholded at 0.9.
• Counts or Multiple Objects success rate is 0.00 for all five T2I models in the human evaluation.
• Flux has the highest average human-graded success rate at 51.04% across 27 failure modes, versus 21.09% for SD3 XL.
• Text generation success drops from 0.520 at 3 tokens to 0.136 at 10 tokens and 0.0 at 50 tokens.

Measured effects reported in the paper

Surprising findings

• All five T2I models score 0% on the Counts or Multiple Objects failure mode according to human evaluation.
• VLMs are prone to confirm the prompt when the original text-to-image prompt is shown, and they often explain distorted anatomy as optical illusions.

Limitations

• Only one LLM judge (Llama3-70B) was used, and only open-source VLMs were evaluated; closed-source models may perform better.
• Best LLM and VLM judges are very large, making the pipeline expensive.
• FineGRAIN agrees with humans only 67.4% of the time and performs poorly on subjective categories such as Surreal (>50% divergence).

Experimental setup

Datasets: FineGRAIN prompts (27 failure modes, 25-30 hand-written prompts each)

Benchmarks: FineGRAIN

Metrics: agreement rate with human labels accuracy success rate

Baselines: VQAScore CLIPScore GenAIBench T2I-CompBench++ PartiPrompt DrawBench

Builds on: VQAScore TIFA DSG Gecko

Closest papers by meaning

• Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.91)
• DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.90)
• Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.90)
• SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis (NeurIPS 2026 · similarity 0.89)
• PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models (CVPR 2025 · similarity 0.89)
• Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)