Research map / Vision-language and multimodal / Visual question answering
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Davidsonian Scene Graph (DSG), a dependency-aware DAG of atomic yes/no questions generated by LLMs to make QG/A text-to-image evaluation more reliable, and releases the DSG-1k benchmark of 1,060 prompts.
Problem
QG/A-based evaluation of text-to-image alignment is unreliable: generated questions are non-atomic, duplicated, hallucinated or omitted, and VQA models answer dependent questions even when their parent premise is false.
Approach
Represent prompt semantics as a DAG of atomic tuples (entities, attributes, relations, globals) inspired by Davidsonian semantics; generate questions from tuples with LLMs in a three-step pipeline, and zero out child questions when a parent question is answered no.
Key results
- DSG questions with PaLI reach Spearman 0.563 / Kendall 0.458 correlation with human Likert scores on TIFA160, beating TIFA (0.431/0.323) and VQ2A (0.207/0.157) with PaLI.
- DSG question generation achieves 92.2% manual precision and 100% recall on 30 TIFA160 prompts, and 98.3% precision / 96.0% recall automatically.
- Per-question VQA-human matching accuracy reaches 73.8% for PaLI, with the model ranking Imagen* ~ MUSE* > SD v2.1 matching human rankings, though PaLI is overly optimistic (~80% vs ~60%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use atomic, unique DSG questions instead of TIFA questions | TIFA160 question generation · TIFA160 (30 samples, manual) | atomicity: 78.5 -> 96.5 | Helped |
| Use atomic, unique DSG questions instead of VQ²A questions | TIFA160 question generation · TIFA160 (30 samples, manual) | atomicity: 3.9 -> 96.5 | Helped |
| Use DSG question generation instead of TIFA | TIFA160 · TIFA160 (160; GPT-3.5) | uniqueness: 73.4 -> 90.5 | Helped |
| Use DSG question generation instead of VQ²A | TIFA160 · TIFA160 (160; GPT-3.5) | uniqueness: 79.8 -> 90.5 | Helped |
| Use DSG questions with PaLI rather than other tested QG/VQA combinations | TIFA160 per-item evaluation · TIFA160 | Spearman's ρ / Kendall's τ: 0.571/0.458 | Helped |
| Use PaLI rather than mPLUG-large for DSG question answering | DSG-1k · DSG-1k | VQA-human match accuracy (%): 72.1 -> 73.8 | Helped |
| Drop dependent questions whose parent answer is NO instead of counting them as NO (ablation) | DSG dependency handling with PaLI · TIFA160 | Spearman's ρ / Kendall's τ: 0.571/0.458 -> 0.561/0.451 | No clear effect |
| Ignore dependency annotations (ablation) | DSG dependency handling with PaLI · TIFA160 | Spearman's ρ / Kendall's τ: 0.571/0.458 -> 0.570/0.457 | No clear effect |
| Drop dependent questions whose parent answer is NO instead of counting them as NO (ablation) | DSG dependency handling with PaLI · DSG-1k | VQA-human matching accuracy: 73.8 -> 71.4 | Hurt |
| Ignore dependency annotations (ablation) | DSG dependency handling with PaLI · DSG-1k | VQA-human matching accuracy: 73.8 -> 65.2 | Hurt |
| Use object detection for counting instead of direct VQA (ablation) | mPLUG-large, Instruct-BLIP, and PaLI · CountBench | VQA-human matching accuracy (%): -1.6, -1.3, -1.2 | Hurt |
| Use OCR for text rendering instead of direct VQA (ablation) | mPLUG-large, Instruct-BLIP, and PaLI · DrawText | VQA-human matching accuracy (%): +4.2, +6.5, +6.5 | Helped |
Surprising findings
- PaLI does not handle text rendering better than mPLUG-large despite its size advantage; text rendering matching ratios are below 40%.
- PaLI's VQA accuracy is overly optimistic relative to human answers (~80% vs ~60% for T2I model accuracy), even though rankings agree.
Limitations
- PaLM 2 and PaLI are not publicly released, limiting reproducibility of the main pipeline.
- Human evaluation results may not be exactly reproduced with different raters; subjective and domain-knowledge questions cause VQA-human disagreement.
- VQA models remain unreliable on counting, text rendering, and abstract attributes such as style and material.
Experimental setup
Datasets: TIFA160 Stanford paragraphs Localized Narratives CountBench VRD DiffusionDB Midjourney-prompts PoseScript Whoops DrawText-Creative
Benchmarks: DSG-1k TIFA160
Metrics: precision recall atomicity uniqueness Spearman rho Kendall tau VQA-human match accuracy
Baselines: TIFA VQ2A CLIPScore BLEU-4 ROUGE-L METEOR SPICE
Builds on: TIFA VQ2A Davidsonian semantics VPEval
Closest papers by meaning
- Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment (ICLR 2025 · similarity 0.90)
- ConVQG: Contrastive Visual Question Generation with Multimodal Guidance (AAAI 2024 · similarity 0.89)
- Image Content Generation with Causal Reasoning (AAAI 2024 · similarity 0.88)
- MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning (NeurIPS 2025 · similarity 0.88)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.88)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.