Research map / Vision-language and multimodal / Multimodal LLMs

PoSh: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PoSh, an open-weight reference-based metric that uses dependency-parsed, coreference-merged scene graphs as structured rubrics for a Qwen3-14B LLM judge, plus the DOCENT artwork description benchmark with granular and coarse human judgments.

Problem

Evaluating long, detailed image descriptions: standard metrics (CIDEr, SPICE) were built for short captions, cannot localize errors to text spans, and GPT-4o-as-a-judge is costly and not replicable.

Approach

Extract attribute- and relation-preserving scene graphs from generation and reference (dependency parsing plus coreference), then verify each component's presence in the other text via templated yes/no-style questions answered by an open LLM, yielding span-level granular scores aggregated into mistake, omission and overall coarse scores.

Key results

• PoSh achieves Spearman rho of 0.519 (mistakes), 0.581 (omissions) and 0.599 (overall quality) on DOCENT, beating the best replicable baseline and GPT-4o in all settings.
• On granular localization, PoSh reaches F1 0.580 for mistakes and 0.680 for omissions, the highest among compared methods.
• PoSh-tuned Qwen2.5-VL-7B via DAPO improves omissions (+0.432) and overall quality (+0.135) over SFT, at the cost of more mistakes (-0.243).

Surprising findings

• The Davidsonian Scene Graph metric performs near chance on DOCENT, with accuracies and correlations near zero.
• Even the best model, GPT-4o, covers only about 50.1% of reference visual information in DOCENT.

Limitations

• LLaVA-Critic (72B) outperforms PoSh on CapArena caption-level and model-level correlations, attributed partly to simple images.
• Coarse human agreement is moderate (Krippendorff alpha 0.409 to 0.509) and the granular judgments cover only 75 images.
• Reward-function experiment is small: 1,000 training images and only 40 pairs judged by graduate students.

Experimental setup

Datasets: DOCENT CapArena DCI DOCCI DetailCaps

Benchmarks: DOCENT CapArena

Metrics: pairwise accuracy Spearman rho Kendall tau macro F1

Baselines: BLEU-4 ROUGE-L METEOR CIDEr SPICE CAPTURE CLIPScore FLEUR Prometheus LLaVA-Critic DCScore Qwen3

Builds on: SPICE Davidsonian Scene Graph CAPTURE DCScore DAPO

Closest papers by meaning

• ContextRef: Evaluating Referenceless Metrics for Image Description Generation (ICLR 2024 · similarity 0.87)
• VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.87)
• Image2Struct: Benchmarking Structure Extraction for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
• Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment (AAAI 2026 · similarity 0.86)
• VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations (ICLR 2026 · similarity 0.86)
• ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.86)